C# DQPublisher Manager


Mathias Kim
 

Hi guys,

 

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.

We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.

How does this generally work?

 

Hope you can help us.

Best regards

 

Mathias Kim

 

ETF Analyst
 
Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim@...
www.crossflow.de
 

Crossflow Financial Advisors GmbH
Sonnenstraße 19
D-80331 München
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner
 

 

The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

 


Ian Bell
 

Hi

Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback. As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk

-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@crossflow.de]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager

Hi guys,

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?

Hope you can help us.
Best regards

Mathias Kim

ETF Analyst

Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>

Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner



The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>


Mathias Kim
 

Hi guys,

we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
- Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
- Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
- We initially thought that openmama/qpid just sends out (broadcasts) messages and doesn't care whether they arrive or not. Is this the case?

Regards

Mathias

-----Original Message-----
From: Ian Bell [mailto:ian.bell@hapsoftconsulting.co.uk]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@lists.openmama.org
Subject: RE: [Openmama-dev] C# DQPublisher Manager

Hi

Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback. As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk


-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@crossflow.de]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager

Hi guys,

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?

Hope you can help us.
Best regards

Mathias Kim

ETF Analyst

Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>

Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner



The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>


Frank Quinn <fquinn.ni@...>
 

Hi Mathias,

I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure.

In fact I just checked and we raised this with the proton guys 2 years ago and they never responded so maybe we should chase that up :) http://qpid.2158936.n2.nabble.com/Detecting-and-capturing-underlying-problems-and-event-with-pn-messenger-td7595635.html

To answer your more specific questions:
 - This is 100% qpid proton / MAMA - not your application's fault
 - The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this
 - This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will always be a little more interactive by nature

On a side note, have you tried our test Qpid Proton Broker implementation http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.

Cheers,
Frank

On Mon, Jun 1, 2015 at 4:00 PM, Mathias Kim <Mathias.Kim@...> wrote:
Hi guys,

we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
-       Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
-       Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
-       We initially thought that openmama/qpid just sends out (broadcasts) messages  and doesn't care whether they arrive or not. Is this the case?

Regards

Mathias

-----Original Message-----
From: Ian Bell [mailto:ian.bell@...]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@...
Subject: RE: [Openmama-dev] C# DQPublisher Manager

Hi

Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback.  As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk


-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@...]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager

Hi guys,

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?

Hope you can help us.
Best regards

Mathias Kim

ETF Analyst

Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>

Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner



The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>

_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev


Frank Quinn <fquinn.ni@...>
 

Hi Mathias,

 

Tracking this via http://bugs.openmama.org/show_bug.cgi?id=41 – looks like the proton folks added something in 0.8 to allow you to pull out link information from a messenger which we may be able to take advantage of. It looks like it’ll involve a little expansion of the endpoint code to store more information about an endpoint than just its URI but it should be doable.

 

Cheers,

Frank


On Mon, Jun 1, 2015 at 7:03 PM, Frank Quinn <fquinn.ni@...> wrote:
Hi Mathias,

I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure.

In fact I just checked and we raised this with the proton guys 2 years ago and they never responded so maybe we should chase that up :) http://qpid.2158936.n2.nabble.com/Detecting-and-capturing-underlying-problems-and-event-with-pn-messenger-td7595635.html

To answer your more specific questions:
 - This is 100% qpid proton / MAMA - not your application's fault
 - The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this
 - This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will always be a little more interactive by nature

On a side note, have you tried our test Qpid Proton Broker implementation http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.

Cheers,
Frank

On Mon, Jun 1, 2015 at 4:00 PM, Mathias Kim <Mathias.Kim@...> wrote:
Hi guys,

we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
-       Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
-       Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
-       We initially thought that openmama/qpid just sends out (broadcasts) messages  and doesn't care whether they arrive or not. Is this the case?

Regards

Mathias

-----Original Message-----
From: Ian Bell [mailto:ian.bell@...]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@...
Subject: RE: [Openmama-dev] C# DQPublisher Manager

Hi

Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback.  As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk


-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@...]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager

Hi guys,

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?

Hope you can help us.
Best regards

Mathias Kim

ETF Analyst

Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>

Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner



The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>

_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev



Mathias Kim
 

Hi Frank,

 

thanks for looking into this.

We’re now having a look into the qpid bridge with broker. We compiled the bridge but then got stuck using it.

Can you provide documentation on how to use the broker feature with the examples, e.g. capturereplayc?

 

Regards

 

Mathias

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Freitag, 5. Juni 2015 00:39
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

 

Tracking this via http://bugs.openmama.org/show_bug.cgi?id=41 – looks like the proton folks added something in 0.8 to allow you to pull out link information from a messenger which we may be able to take advantage of. It looks like it’ll involve a little expansion of the endpoint code to store more information about an endpoint than just its URI but it should be doable.

 

Cheers,

Frank

 

On Mon, Jun 1, 2015 at 7:03 PM, Frank Quinn <fquinn.ni@...> wrote:

Hi Mathias,

I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure.

In fact I just checked and we raised this with the proton guys 2 years ago and they never responded so maybe we should chase that up :) http://qpid.2158936.n2.nabble.com/Detecting-and-capturing-underlying-problems-and-event-with-pn-messenger-td7595635.html

To answer your more specific questions:

 - This is 100% qpid proton / MAMA - not your application's fault

 - The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this

 - This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will always be a little more interactive by nature

On a side note, have you tried our test Qpid Proton Broker implementation http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.

 

Cheers,

Frank

 

On Mon, Jun 1, 2015 at 4:00 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi guys,

we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
-       Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
-       Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
-       We initially thought that openmama/qpid just sends out (broadcasts) messages  and doesn't care whether they arrive or not. Is this the case?

Regards

Mathias


-----Original Message-----
From: Ian Bell [mailto:ian.bell@...]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@...
Subject: RE: [Openmama-dev] C# DQPublisher Manager

Hi

Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback.  As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk


-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@...]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager

Hi guys,

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?

Hope you can help us.
Best regards

Mathias Kim

ETF Analyst

Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>

Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner



The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>

_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

 


Frank Quinn <fquinn.ni@...>
 

Hi Mathias,

The best documentation for this was probably on the original commit note:


It contains an example transport - all you need to to is change your IPs to the IP address of qpidd (running as described) and then just run your MAMA example applications using -m qpid -tport qpidbroker.

Note it contains a new mama.qpid.transport.<transportname>.type parameter to let the bridge know that this is a broker transport.

Cheers,
Frank

On Fri, Jun 5, 2015 at 1:20 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

thanks for looking into this.

We’re now having a look into the qpid bridge with broker. We compiled the bridge but then got stuck using it.

Can you provide documentation on how to use the broker feature with the examples, e.g. capturereplayc?

 

Regards

 

Mathias

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Freitag, 5. Juni 2015 00:39
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

 

Tracking this via http://bugs.openmama.org/show_bug.cgi?id=41 – looks like the proton folks added something in 0.8 to allow you to pull out link information from a messenger which we may be able to take advantage of. It looks like it’ll involve a little expansion of the endpoint code to store more information about an endpoint than just its URI but it should be doable.

 

Cheers,

Frank

 

On Mon, Jun 1, 2015 at 7:03 PM, Frank Quinn <fquinn.ni@...> wrote:

Hi Mathias,

I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure.

In fact I just checked and we raised this with the proton guys 2 years ago and they never responded so maybe we should chase that up :) http://qpid.2158936.n2.nabble.com/Detecting-and-capturing-underlying-problems-and-event-with-pn-messenger-td7595635.html

To answer your more specific questions:

 - This is 100% qpid proton / MAMA - not your application's fault

 - The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this

 - This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will always be a little more interactive by nature

On a side note, have you tried our test Qpid Proton Broker implementation http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.

 

Cheers,

Frank

 

On Mon, Jun 1, 2015 at 4:00 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi guys,

we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
-       Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
-       Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
-       We initially thought that openmama/qpid just sends out (broadcasts) messages  and doesn't care whether they arrive or not. Is this the case?

Regards

Mathias


-----Original Message-----
From: Ian Bell [mailto:ian.bell@...]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@...
Subject: RE: [Openmama-dev] C# DQPublisher Manager

Hi

Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback.  As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk


-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@...]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager

Hi guys,

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?

Hope you can help us.
Best regards

Mathias Kim

ETF Analyst

Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>

Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner



The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>

_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

 



Frank Quinn <fquinn.ni@...>
 

Oh - and with the broker implementation, you no longer require a separate transport for pub and sub unless they're on different subnets - you just point both sides to the broker.

Cheers,
Frank

On Fri, Jun 5, 2015 at 1:32 PM, Frank Quinn <fquinn.ni@...> wrote:
Hi Mathias,

The best documentation for this was probably on the original commit note:


It contains an example transport - all you need to to is change your IPs to the IP address of qpidd (running as described) and then just run your MAMA example applications using -m qpid -tport qpidbroker.

Note it contains a new mama.qpid.transport.<transportname>.type parameter to let the bridge know that this is a broker transport.

Cheers,
Frank

On Fri, Jun 5, 2015 at 1:20 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

thanks for looking into this.

We’re now having a look into the qpid bridge with broker. We compiled the bridge but then got stuck using it.

Can you provide documentation on how to use the broker feature with the examples, e.g. capturereplayc?

 

Regards

 

Mathias

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Freitag, 5. Juni 2015 00:39
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

 

Tracking this via http://bugs.openmama.org/show_bug.cgi?id=41 – looks like the proton folks added something in 0.8 to allow you to pull out link information from a messenger which we may be able to take advantage of. It looks like it’ll involve a little expansion of the endpoint code to store more information about an endpoint than just its URI but it should be doable.

 

Cheers,

Frank

 

On Mon, Jun 1, 2015 at 7:03 PM, Frank Quinn <fquinn.ni@...> wrote:

Hi Mathias,

I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure.

In fact I just checked and we raised this with the proton guys 2 years ago and they never responded so maybe we should chase that up :) http://qpid.2158936.n2.nabble.com/Detecting-and-capturing-underlying-problems-and-event-with-pn-messenger-td7595635.html

To answer your more specific questions:

 - This is 100% qpid proton / MAMA - not your application's fault

 - The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this

 - This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will always be a little more interactive by nature

On a side note, have you tried our test Qpid Proton Broker implementation http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.

 

Cheers,

Frank

 

On Mon, Jun 1, 2015 at 4:00 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi guys,

we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
-       Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
-       Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
-       We initially thought that openmama/qpid just sends out (broadcasts) messages  and doesn't care whether they arrive or not. Is this the case?

Regards

Mathias


-----Original Message-----
From: Ian Bell [mailto:ian.bell@...]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@...
Subject: RE: [Openmama-dev] C# DQPublisher Manager

Hi

Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback.  As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk


-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@...]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager

Hi guys,

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?

Hope you can help us.
Best regards

Mathias Kim

ETF Analyst

Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>

Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner



The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>

_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

 




Mathias Kim
 

Hi Frank,

 

Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as it seems on Linux.

As you suggested when using the broker the slowdown after client disconnect disappears. However, the broker seems to be very slow.

 

We were testing it with the same C# classes that showed acceptable performance without the broker. Using the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.

Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available cores evenly.

 

How is the performance on Linux?

 

We’re using

-          Windows 8

-          Broker, publisher and client on the same machine

-          Qpid 0.32 & Proton 0.7

 

Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without, no communication happens.

 

Regards

 

Mathias


Frank Quinn <fquinn.ni@...>
 

Hi Mathias,

I just tried an equivalent test with Qpid Broker 0.32 and proton 0.9.1 (with a few small changes to let it compile as proton changed their interface again) and the producer keeps crashing... so we'll get back to what's going on there later.

So I tried with Proton 0.7 on linux (Fedora 22) and I can see:

1. Logging doesn't need to be disabled - seems to work regardless.
2. Performance is pretty much the same with the broker as it is point to point
3. CPU is spread across several cores
4. Latency is around 4ms at data rates of around 10,000 msg/s

If you're getting latencies in the 2 second region, my first guess is that you're queuing in your client, have you tried printing out the queue size in your application? This is all finger in the air stuff though because this is all running on my local laptop.

When you're operating point to point, the producer is essentially throttled by the speed of the consumer. When a broker is involved, this relationship is effectively decoupled, so you may find that the producer is actually publishing faster than you expect now (enqueuing for sending is not the same as actually sending). That or else your applications haven't been compiled with multithreading support and they're all just fighting with each other for scheduler time. I'll try a windows test when I get a chance to see if I can replicate what you're seeing.

What sort of data rates and message sizes were you putting through?

Cheers,
Frank

On Tue, Jun 16, 2015 at 4:31 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as it seems on Linux.

As you suggested when using the broker the slowdown after client disconnect disappears. However, the broker seems to be very slow.

 

We were testing it with the same C# classes that showed acceptable performance without the broker. Using the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.

Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available cores evenly.

 

How is the performance on Linux?

 

We’re using

-          Windows 8

-          Broker, publisher and client on the same machine

-          Qpid 0.32 & Proton 0.7

 

Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without, no communication happens.

 

Regards

 

Mathias



Mathias Kim
 

Hi Frank,

 

thanks for the answer.

 

I tried using Proton 0.8 but that changed nothing.

Also, I checked the queue depth. Publisher and Subscriber app had a depth of 0.

In addition, I checked the broker with capturereplayc and mamalistenc. More or less the same results: with one symbol there was no visible latency. After subscribing to 5 symbols I saw the same behavior as with our C# Publisher/Subscriber. So I think the issue lies with the Windows Broker. Also, I was running it on a Server within my network and the delay got even larger.

So I think the delay is caused by the (Windows) Qpid broker.

 

Since I have no clue how to modify the Windows Qpid broker, I’m going to test it with a Fedora Virtual Machine.

 

Regards,

 

Mathias

 

 

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Samstag, 20. Juni 2015 13:09
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

I just tried an equivalent test with Qpid Broker 0.32 and proton 0.9.1 (with a few small changes to let it compile as proton changed their interface again) and the producer keeps crashing... so we'll get back to what's going on there later.

So I tried with Proton 0.7 on linux (Fedora 22) and I can see:

1. Logging doesn't need to be disabled - seems to work regardless.

2. Performance is pretty much the same with the broker as it is point to point

3. CPU is spread across several cores

4. Latency is around 4ms at data rates of around 10,000 msg/s

If you're getting latencies in the 2 second region, my first guess is that you're queuing in your client, have you tried printing out the queue size in your application? This is all finger in the air stuff though because this is all running on my local laptop.

When you're operating point to point, the producer is essentially throttled by the speed of the consumer. When a broker is involved, this relationship is effectively decoupled, so you may find that the producer is actually publishing faster than you expect now (enqueuing for sending is not the same as actually sending). That or else your applications haven't been compiled with multithreading support and they're all just fighting with each other for scheduler time. I'll try a windows test when I get a chance to see if I can replicate what you're seeing.

What sort of data rates and message sizes were you putting through?

 

Cheers,

Frank

 

On Tue, Jun 16, 2015 at 4:31 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as it seems on Linux.

As you suggested when using the broker the slowdown after client disconnect disappears. However, the broker seems to be very slow.

 

We were testing it with the same C# classes that showed acceptable performance without the broker. Using the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.

Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available cores evenly.

 

How is the performance on Linux?

 

We’re using

-          Windows 8

-          Broker, publisher and client on the same machine

-          Qpid 0.32 & Proton 0.7

 

Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without, no communication happens.

 

Regards

 

Mathias

 


Frank Quinn <fquinn.ni@...>
 

Hi Mathias,

I was only testing with producer / consumer which is only one symbol so perhaps that's part of the issue here. I'll test with more symbols too when I get back to my machine. What sort of data rates were you running at?

Cheers,
Frank

On Mon, Jun 22, 2015 at 4:25 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

thanks for the answer.

 

I tried using Proton 0.8 but that changed nothing.

Also, I checked the queue depth. Publisher and Subscriber app had a depth of 0.

In addition, I checked the broker with capturereplayc and mamalistenc. More or less the same results: with one symbol there was no visible latency. After subscribing to 5 symbols I saw the same behavior as with our C# Publisher/Subscriber. So I think the issue lies with the Windows Broker. Also, I was running it on a Server within my network and the delay got even larger.

So I think the delay is caused by the (Windows) Qpid broker.

 

Since I have no clue how to modify the Windows Qpid broker, I’m going to test it with a Fedora Virtual Machine.

 

Regards,

 

Mathias

 

 

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Samstag, 20. Juni 2015 13:09
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

I just tried an equivalent test with Qpid Broker 0.32 and proton 0.9.1 (with a few small changes to let it compile as proton changed their interface again) and the producer keeps crashing... so we'll get back to what's going on there later.

So I tried with Proton 0.7 on linux (Fedora 22) and I can see:

1. Logging doesn't need to be disabled - seems to work regardless.

2. Performance is pretty much the same with the broker as it is point to point

3. CPU is spread across several cores

4. Latency is around 4ms at data rates of around 10,000 msg/s

If you're getting latencies in the 2 second region, my first guess is that you're queuing in your client, have you tried printing out the queue size in your application? This is all finger in the air stuff though because this is all running on my local laptop.

When you're operating point to point, the producer is essentially throttled by the speed of the consumer. When a broker is involved, this relationship is effectively decoupled, so you may find that the producer is actually publishing faster than you expect now (enqueuing for sending is not the same as actually sending). That or else your applications haven't been compiled with multithreading support and they're all just fighting with each other for scheduler time. I'll try a windows test when I get a chance to see if I can replicate what you're seeing.

What sort of data rates and message sizes were you putting through?

 

Cheers,

Frank

 

On Tue, Jun 16, 2015 at 4:31 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as it seems on Linux.

As you suggested when using the broker the slowdown after client disconnect disappears. However, the broker seems to be very slow.

 

We were testing it with the same C# classes that showed acceptable performance without the broker. Using the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.

Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available cores evenly.

 

How is the performance on Linux?

 

We’re using

-          Windows 8

-          Broker, publisher and client on the same machine

-          Qpid 0.32 & Proton 0.7

 

Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without, no communication happens.

 

Regards

 

Mathias

 



Mathias Kim
 

Hi Frank,

 

I tested the broker on a Fedora Virtualbox and the results were much better than on my Windows version. The performance was as good as with point to point for capturereplayc/mamalistenc (5 Symbols) and our C# classes.

For testing the C# classes I was simulating orderbooks. In the end I was sending an average of 14 messages per second. Each message contained on average 25 fields.

The same test with a Windows broker led to huge delay. Since the amount of data is fairly low, something must have gone wrong when compiling qpid under Windows.

Interestingly, also under Fedora I have to add the –t parameter. Without, no communication happens.

 

Unfortunately, with Virtualbox/Fedora our C# publisher and listener experience some problems since not every subscription is successful. In about 50% of the cases I get this behavior:

-          Publisher connects successfully to qpid broker

-          Listener connects successfully to qpid broker and submits desired symbol

-          Symbol arrives at publisher, publisher starts sending data to qpid broker

-          Qpid broker receives data (as I can see on the output) but never forwards to the listener

Monitoring the port shows that no messages ever arrive at the machine the listener is running on.

No similar behavior with capturereplayc/mamalistenc. Also, no similar issues with qpid under Windows.

 

Did you test the qpid broker/openmama combo with a C# Listener yet?

 

Regards

 

Mathias

 

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Montag, 22. Juni 2015 17:40
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

 

I was only testing with producer / consumer which is only one symbol so perhaps that's part of the issue here. I'll test with more symbols too when I get back to my machine. What sort of data rates were you running at?

 

Cheers,

Frank

 

On Mon, Jun 22, 2015 at 4:25 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

thanks for the answer.

 

I tried using Proton 0.8 but that changed nothing.

Also, I checked the queue depth. Publisher and Subscriber app had a depth of 0.

In addition, I checked the broker with capturereplayc and mamalistenc. More or less the same results: with one symbol there was no visible latency. After subscribing to 5 symbols I saw the same behavior as with our C# Publisher/Subscriber. So I think the issue lies with the Windows Broker. Also, I was running it on a Server within my network and the delay got even larger.

So I think the delay is caused by the (Windows) Qpid broker.

 

Since I have no clue how to modify the Windows Qpid broker, I’m going to test it with a Fedora Virtual Machine.

 

Regards,

 

Mathias

 

 

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Samstag, 20. Juni 2015 13:09
To: Mathias Kim
Cc:
openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

I just tried an equivalent test with Qpid Broker 0.32 and proton 0.9.1 (with a few small changes to let it compile as proton changed their interface again) and the producer keeps crashing... so we'll get back to what's going on there later.

So I tried with Proton 0.7 on linux (Fedora 22) and I can see:

1. Logging doesn't need to be disabled - seems to work regardless.

2. Performance is pretty much the same with the broker as it is point to point

3. CPU is spread across several cores

4. Latency is around 4ms at data rates of around 10,000 msg/s

If you're getting latencies in the 2 second region, my first guess is that you're queuing in your client, have you tried printing out the queue size in your application? This is all finger in the air stuff though because this is all running on my local laptop.

When you're operating point to point, the producer is essentially throttled by the speed of the consumer. When a broker is involved, this relationship is effectively decoupled, so you may find that the producer is actually publishing faster than you expect now (enqueuing for sending is not the same as actually sending). That or else your applications haven't been compiled with multithreading support and they're all just fighting with each other for scheduler time. I'll try a windows test when I get a chance to see if I can replicate what you're seeing.

What sort of data rates and message sizes were you putting through?

 

Cheers,

Frank

 

On Tue, Jun 16, 2015 at 4:31 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as it seems on Linux.

As you suggested when using the broker the slowdown after client disconnect disappears. However, the broker seems to be very slow.

 

We were testing it with the same C# classes that showed acceptable performance without the broker. Using the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.

Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available cores evenly.

 

How is the performance on Linux?

 

We’re using

-          Windows 8

-          Broker, publisher and client on the same machine

-          Qpid 0.32 & Proton 0.7

 

Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without, no communication happens.

 

Regards

 

Mathias