|
Hi guys,
we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain
symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us
on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?
Hope you can help us.
Best regards
Mathias Kim
ETF Analyst
Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim@...
www.crossflow.de
Crossflow Financial Advisors GmbH
Sonnenstraße 19
D-80331 München
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner
The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction,
disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free,
as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.
|
|
|
Hi
Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback. As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.
Thanks,
Ian Bell
Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk
toggle quoted message
Show quoted text
-----Original Message----- From: Mathias Kim [mailto:Mathias.Kim@...] Sent: 25 March 2015 16:40 Subject: [Openmama-dev] C# DQPublisher Manager Hi guys, we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol. We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves. How does this generally work? Hope you can help us. Best regards Mathias Kim ETF Analyst Phone +49 89 442327-191 Mobile +49 171 4152665 mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de< https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de> Crossflow Financial Advisors GmbH Sonnenstra?e 19 D-80331 M?nchen Local Court of Munich HRB 186408 Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete. -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>
|
|
|
Hi guys,
we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
- Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
- Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
- We initially thought that openmama/qpid just sends out (broadcasts) messages and doesn't care whether they arrive or not. Is this the case?
Regards
Mathias
toggle quoted message
Show quoted text
-----Original Message----- From: Ian Bell [mailto:ian.bell@...] Sent: Dienstag, 14. April 2015 13:49 To: Mathias Kim Cc: openmama-dev@... Subject: RE: [Openmama-dev] C# DQPublisher Manager Hi Just wondering if you go this sorted? The first request should route through to the OnNewRequst callback. As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely. Can you supply the code and some logs and I may be able to spot the issue. Thanks, Ian Bell Hapsoft Consulting Ltd 138 Belmont Road, Belfast, BT4 2AQ Office: +447481866051 Web: www.hapsoftconsulting.co.uk -----Original Message----- From: Mathias Kim [mailto:Mathias.Kim@...] Sent: 25 March 2015 16:40 Subject: [Openmama-dev] C# DQPublisher Manager Hi guys, we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol. We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves. How does this generally work? Hope you can help us. Best regards Mathias Kim ETF Analyst Phone +49 89 442327-191 Mobile +49 171 4152665 mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de< https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de> Crossflow Financial Advisors GmbH Sonnenstra?e 19 D-80331 M?nchen Local Court of Munich HRB 186408 Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete. -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>
|
|
|
Frank Quinn <fquinn.ni@...>
Hi Mathias,
I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure. To answer your more specific questions:
- This is 100% qpid proton / MAMA - not your application's fault
- The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this
- This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will always be a little more interactive by nature
On a side note, have you tried our test Qpid Proton Broker implementation http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.
Cheers, Frank
On Mon, Jun 1, 2015 at 4:00 PM, Mathias Kim <Mathias.Kim@...> wrote: Hi guys,
we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
- Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
- Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
- We initially thought that openmama/qpid just sends out (broadcasts) messages and doesn't care whether they arrive or not. Is this the case?
Regards
Mathias
-----Original Message-----
From: Ian Bell [mailto: ian.bell@...]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@...
Subject: RE: [Openmama-dev] C# DQPublisher Manager
Hi
Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback. As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.
Thanks,
Ian Bell
Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk
-----Original Message-----
From: Mathias Kim [mailto: Mathias.Kim@...]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager
Hi guys,
we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?
Hope you can help us.
Best regards
Mathias Kim
ETF Analyst
Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto: mathias.kim at crossflow.de> www.crossflow.de< https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>
Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner
The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: < http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>
_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev
|
|
|
Frank Quinn <fquinn.ni@...>
Hi Mathias,
Tracking this via http://bugs.openmama.org/show_bug.cgi?id=41
– looks like the proton folks added something in 0.8 to allow you to pull out link information from a messenger which we may be able to take advantage
of. It looks like it’ll involve a little expansion of the endpoint code
to store more information about an endpoint than just its URI but it should be doable.
Cheers,
Frank
On Mon, Jun 1, 2015 at 7:03 PM, Frank Quinn <fquinn.ni@...> wrote: Hi Mathias,
I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure. To answer your more specific questions:
- This is 100% qpid proton / MAMA - not your application's fault
- The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this
- This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will always be a little more interactive by nature
On a side note, have you tried our test Qpid Proton Broker implementation http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.
Cheers, Frank
|
|
|
Hi Frank,
thanks for looking into this.
We’re now having a look into the qpid bridge with broker. We compiled the bridge but then got stuck using it.
Can you provide documentation on how to use the broker feature with the examples, e.g. capturereplayc?
Regards
Mathias
toggle quoted message
Show quoted text
From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Freitag, 5. Juni 2015 00:39
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager
Hi Mathias,
Tracking this via
http://bugs.openmama.org/show_bug.cgi?id=41 – looks like the proton folks added something in 0.8 to allow you to pull out link information from a messenger which we may be able to take advantage of.
It looks like it’ll involve a little expansion of the endpoint code to store more information about an endpoint than just its URI but it should be doable.
Cheers,
Frank
On Mon, Jun 1, 2015 at 7:03 PM, Frank Quinn <fquinn.ni@...> wrote:
I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest
releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you
look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure.
To answer your more specific questions:
- This is 100% qpid proton / MAMA - not your application's fault
- The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this
- This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will
always be a little more interactive by nature
On a side note, have you tried our test Qpid Proton Broker implementation
http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for
example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.
Cheers,
Frank
On Mon, Jun 1, 2015 at 4:00 PM, Mathias Kim <Mathias.Kim@...> wrote:
Hi guys,
we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends
on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is
done, the server is reacting normal and works as before.
We now ask ourselves:
- Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
- Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
- We initially thought that openmama/qpid just sends out (broadcasts) messages and doesn't care whether they arrive or not. Is this the case?
Regards
Mathias
-----Original Message-----
From: Ian Bell [mailto:ian.bell@...]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@...
Subject: RE: [Openmama-dev] C# DQPublisher Manager
Hi
Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback. As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.
Thanks,
Ian Bell
Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk
-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@...]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager
Hi guys,
we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that
we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?
Hope you can help us.
Best regards
Mathias Kim
ETF Analyst
Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at
crossflow.de>
www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>
Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner
The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify
the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>
_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev
|
|
|
Frank Quinn <fquinn.ni@...>
Hi Mathias,
The best documentation for this was probably on the original commit note:
It contains an example transport - all you need to to is change your IPs to the IP address of qpidd (running as described) and then just run your MAMA example applications using -m qpid -tport qpidbroker.
Note it contains a new mama.qpid.transport.<transportname>.type parameter to let the bridge know that this is a broker transport.
Cheers, Frank
On Fri, Jun 5, 2015 at 1:20 PM, Mathias Kim <Mathias.Kim@...> wrote:
Hi Frank,
thanks for looking into this.
We’re now having a look into the qpid bridge with broker. We compiled the bridge but then got stuck using it.
Can you provide documentation on how to use the broker feature with the examples, e.g. capturereplayc?
Regards
Mathias
From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Freitag, 5. Juni 2015 00:39
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager
Hi Mathias,
Tracking this via
http://bugs.openmama.org/show_bug.cgi?id=41 – looks like the proton folks added something in 0.8 to allow you to pull out link information from a messenger which we may be able to take advantage of.
It looks like it’ll involve a little expansion of the endpoint code to store more information about an endpoint than just its URI but it should be doable.
Cheers,
Frank
On Mon, Jun 1, 2015 at 7:03 PM, Frank Quinn <fquinn.ni@...> wrote:
I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest
releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you
look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure.
To answer your more specific questions:
- This is 100% qpid proton / MAMA - not your application's fault
- The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this
- This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will
always be a little more interactive by nature
On a side note, have you tried our test Qpid Proton Broker implementation
http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for
example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.
Cheers,
Frank
On Mon, Jun 1, 2015 at 4:00 PM, Mathias Kim <Mathias.Kim@...> wrote:
Hi guys,
we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends
on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is
done, the server is reacting normal and works as before.
We now ask ourselves:
- Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
- Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
- We initially thought that openmama/qpid just sends out (broadcasts) messages and doesn't care whether they arrive or not. Is this the case?
Regards
Mathias
-----Original Message-----
From: Ian Bell [mailto:ian.bell@...]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@...
Subject: RE: [Openmama-dev] C# DQPublisher Manager
Hi
Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback. As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.
Thanks,
Ian Bell
Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk
-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@...]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager
Hi guys,
we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that
we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?
Hope you can help us.
Best regards
Mathias Kim
ETF Analyst
Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at
crossflow.de>
www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>
Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner
The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify
the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>
_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev
|
|
|
Frank Quinn <fquinn.ni@...>
Oh - and with the broker implementation, you no longer require a separate transport for pub and sub unless they're on different subnets - you just point both sides to the broker.
Cheers, Frank
On Fri, Jun 5, 2015 at 1:32 PM, Frank Quinn <fquinn.ni@...> wrote: Hi Mathias,
The best documentation for this was probably on the original commit note:
It contains an example transport - all you need to to is change your IPs to the IP address of qpidd (running as described) and then just run your MAMA example applications using -m qpid -tport qpidbroker.
Note it contains a new mama.qpid.transport.<transportname>.type parameter to let the bridge know that this is a broker transport.
Cheers, Frank
|
|
|
Hi Frank,
Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as it seems on Linux.
As you suggested when using the broker the slowdown after client disconnect disappears. However, the broker seems to be very slow.
We were testing it with the same C# classes that showed acceptable performance without the broker. Using the broker, every received message adds a delay of roughly
two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.
Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available cores evenly.
How is the performance on Linux?
We’re using
-
Windows 8
-
Broker, publisher and client on the same machine
-
Qpid 0.32 & Proton 0.7
Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without, no communication happens.
Regards
Mathias
|
|
|
Frank Quinn <fquinn.ni@...>
Hi Mathias,
I just tried an equivalent test with Qpid Broker 0.32 and proton 0.9.1 (with a few small changes to let it compile as proton changed their interface again) and the producer keeps crashing... so we'll get back to what's going on there later. So I tried with Proton 0.7 on linux (Fedora 22) and I can see: 1. Logging doesn't need to be disabled - seems to work regardless. 2. Performance is pretty much the same with the broker as it is point to point 3. CPU is spread across several cores
4. Latency is around 4ms at data rates of around 10,000 msg/s
If you're getting latencies in the 2 second region, my first guess is that you're queuing in your client, have you tried printing out the queue size in your application? This is all finger in the air stuff though because this is all running on my local laptop.
When you're operating point to point, the producer is essentially throttled by the speed of the consumer. When a broker is involved, this relationship is effectively decoupled, so you may find that the producer is actually publishing faster than you expect now (enqueuing for sending is not the same as actually sending). That or else your applications haven't been compiled with multithreading support and they're all just fighting with each other for scheduler time. I'll try a windows test when I get a chance to see if I can replicate what you're seeing.
What sort of data rates and message sizes were you putting through?
Cheers, Frank
On Tue, Jun 16, 2015 at 4:31 PM, Mathias Kim <Mathias.Kim@...> wrote:
Hi Frank,
Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as it seems on Linux.
As you suggested when using the broker the slowdown after client disconnect disappears. However, the broker seems to be very slow.
We were testing it with the same C# classes that showed acceptable performance without the broker. Using the broker, every received message adds a delay of roughly
two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.
Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available cores evenly.
How is the performance on Linux?
We’re using
-
Windows 8
-
Broker, publisher and client on the same machine
-
Qpid 0.32 & Proton 0.7
Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without, no communication happens.
Regards
Mathias
|
|
|
Hi Frank,
thanks for the answer.
I tried using Proton 0.8 but that changed nothing.
Also, I checked the queue depth. Publisher and Subscriber app had a depth of 0.
In addition, I checked the broker with capturereplayc and mamalistenc. More or less the same results: with one symbol
there was no visible latency. After subscribing to 5 symbols I saw the same behavior as with our C# Publisher/Subscriber. So I think the issue lies with the Windows Broker. Also, I was running it on a Server within my network and the delay got even larger.
So I think the delay is caused by the (Windows) Qpid broker.
Since I have no clue how to modify the Windows Qpid broker, I’m going to test it with a Fedora Virtual Machine.
Regards,
Mathias
toggle quoted message
Show quoted text
From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Samstag, 20. Juni 2015 13:09
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager
I just tried an equivalent test with Qpid Broker 0.32 and proton 0.9.1 (with a few small changes to let it compile as proton changed their interface again) and the producer keeps crashing... so we'll get back
to what's going on there later.
So I tried with Proton 0.7 on linux (Fedora 22) and I can see:
1. Logging doesn't need to be disabled - seems to work regardless.
2. Performance is pretty much the same with the broker as it is point to point
3. CPU is spread across several cores
4. Latency is around 4ms at data rates of around 10,000 msg/s
If you're getting latencies in the 2 second region, my first guess is that you're queuing in your client, have you tried printing out the queue size in your application? This is all finger in the air stuff though
because this is all running on my local laptop.
When you're operating point to point, the producer is essentially throttled by the speed of the consumer. When a broker is involved, this relationship is effectively decoupled, so you may find that the producer is actually publishing faster
than you expect now (enqueuing for sending is not the same as actually sending). That or else your applications haven't been compiled with multithreading support and they're all just fighting with each other for scheduler time. I'll try a windows test when
I get a chance to see if I can replicate what you're seeing.
What sort of data rates and message sizes were you putting through?
Cheers,
Frank
On Tue, Jun 16, 2015 at 4:31 PM, Mathias Kim <Mathias.Kim@...> wrote:
Hi Frank,
Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as
it seems on Linux.
As you suggested when using the broker the slowdown after client disconnect disappears. However, the
broker seems to be very slow.
We were testing it with the same C# classes that showed acceptable performance without the broker. Using
the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.
Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available
cores evenly.
How is the performance on Linux?
We’re using
-
Windows 8
-
Broker, publisher and client on the same machine
-
Qpid 0.32 & Proton 0.7
Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without,
no communication happens.
Regards
Mathias
|
|
|
Frank Quinn <fquinn.ni@...>
Hi Mathias,
I was only testing with producer / consumer which is only one symbol so perhaps that's part of the issue here. I'll test with more symbols too when I get back to my machine. What sort of data rates were you running at?
Cheers, Frank
On Mon, Jun 22, 2015 at 4:25 PM, Mathias Kim <Mathias.Kim@...> wrote:
Hi Frank,
thanks for the answer.
I tried using Proton 0.8 but that changed nothing.
Also, I checked the queue depth. Publisher and Subscriber app had a depth of 0.
In addition, I checked the broker with capturereplayc and mamalistenc. More or less the same results: with one symbol
there was no visible latency. After subscribing to 5 symbols I saw the same behavior as with our C# Publisher/Subscriber. So I think the issue lies with the Windows Broker. Also, I was running it on a Server within my network and the delay got even larger.
So I think the delay is caused by the (Windows) Qpid broker.
Since I have no clue how to modify the Windows Qpid broker, I’m going to test it with a Fedora Virtual Machine.
Regards,
Mathias
From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Samstag, 20. Juni 2015 13:09
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager
I just tried an equivalent test with Qpid Broker 0.32 and proton 0.9.1 (with a few small changes to let it compile as proton changed their interface again) and the producer keeps crashing... so we'll get back
to what's going on there later.
So I tried with Proton 0.7 on linux (Fedora 22) and I can see:
1. Logging doesn't need to be disabled - seems to work regardless.
2. Performance is pretty much the same with the broker as it is point to point
3. CPU is spread across several cores
4. Latency is around 4ms at data rates of around 10,000 msg/s
If you're getting latencies in the 2 second region, my first guess is that you're queuing in your client, have you tried printing out the queue size in your application? This is all finger in the air stuff though
because this is all running on my local laptop.
When you're operating point to point, the producer is essentially throttled by the speed of the consumer. When a broker is involved, this relationship is effectively decoupled, so you may find that the producer is actually publishing faster
than you expect now (enqueuing for sending is not the same as actually sending). That or else your applications haven't been compiled with multithreading support and they're all just fighting with each other for scheduler time. I'll try a windows test when
I get a chance to see if I can replicate what you're seeing.
What sort of data rates and message sizes were you putting through?
Cheers,
Frank
On Tue, Jun 16, 2015 at 4:31 PM, Mathias Kim <Mathias.Kim@...> wrote:
Hi Frank,
Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as
it seems on Linux.
As you suggested when using the broker the slowdown after client disconnect disappears. However, the
broker seems to be very slow.
We were testing it with the same C# classes that showed acceptable performance without the broker. Using
the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.
Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available
cores evenly.
How is the performance on Linux?
We’re using
-
Windows 8
-
Broker, publisher and client on the same machine
-
Qpid 0.32 & Proton 0.7
Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without,
no communication happens.
Regards
Mathias
|
|
|
Hi Frank,
I tested the broker on a Fedora Virtualbox and the results were much better than on my Windows version. The performance
was as good as with point to point for capturereplayc/mamalistenc (5 Symbols) and our C# classes.
For testing the C# classes I was simulating orderbooks. In the end I was sending an average of 14 messages per second.
Each message contained on average 25 fields.
The same test with a Windows broker led to huge delay. Since the amount of data is fairly low, something must have gone
wrong when compiling qpid under Windows.
Interestingly, also under Fedora I have to add the –t parameter. Without, no communication happens.
Unfortunately, with Virtualbox/Fedora our C# publisher and listener experience some problems since not every subscription
is successful. In about 50% of the cases I get this behavior:
-
Publisher connects successfully to qpid broker
-
Listener connects successfully to qpid broker and submits desired symbol
-
Symbol arrives at publisher, publisher starts sending data to qpid broker
-
Qpid broker receives data (as I can see on the output) but never forwards to the listener
Monitoring the port shows that no messages ever arrive at the machine the listener is running on.
No similar behavior with capturereplayc/mamalistenc. Also, no similar issues with qpid under Windows.
Did you test the qpid broker/openmama combo with a C# Listener yet?
Regards
Mathias
toggle quoted message
Show quoted text
From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Montag, 22. Juni 2015 17:40
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager
Hi Mathias,
I was only testing with producer / consumer which is only one symbol so perhaps that's part of the issue here. I'll test with more symbols too when I get back to my machine. What sort of data rates were you running at?
On Mon, Jun 22, 2015 at 4:25 PM, Mathias Kim <Mathias.Kim@...> wrote:
Hi Frank,
thanks for the answer.
I tried using Proton 0.8 but that changed nothing.
Also, I checked the queue depth. Publisher and Subscriber app had a depth of 0.
In addition, I checked the broker with capturereplayc and mamalistenc. More or less the
same results: with one symbol there was no visible latency. After subscribing to 5 symbols I saw the same behavior as with our C# Publisher/Subscriber. So I think the issue lies with the Windows Broker. Also, I was running it on a Server within my network
and the delay got even larger.
So I think the delay is caused by the (Windows) Qpid broker.
Since I have no clue how to modify the Windows Qpid broker, I’m going to test it with
a Fedora Virtual Machine.
Regards,
Mathias
From: Frank
Quinn [mailto:fquinn.ni@...]
Sent: Samstag, 20. Juni 2015 13:09
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager
I just tried an equivalent test with Qpid Broker 0.32 and proton 0.9.1 (with a few small changes to let it compile as proton changed their interface again) and the
producer keeps crashing... so we'll get back to what's going on there later.
So I tried with Proton 0.7 on linux (Fedora 22) and I can see:
1. Logging doesn't need to be disabled - seems to work regardless.
2. Performance is pretty much the same with the broker as it is point to point
3. CPU is spread across several cores
4. Latency is around 4ms at data rates of around 10,000 msg/s
If you're getting latencies in the 2 second region, my first guess is that you're queuing in your client, have you tried printing out the queue size in your application?
This is all finger in the air stuff though because this is all running on my local laptop.
When you're operating point to point, the producer is essentially throttled by the speed of the consumer. When a broker is involved, this relationship is effectively
decoupled, so you may find that the producer is actually publishing faster than you expect now (enqueuing for sending is not the same as actually sending). That or else your applications haven't been compiled with multithreading support and they're all just
fighting with each other for scheduler time. I'll try a windows test when I get a chance to see if I can replicate what you're seeing.
What sort of data rates and message sizes were you putting through?
Cheers,
Frank
On Tue, Jun 16, 2015 at 4:31 PM, Mathias Kim <Mathias.Kim@...>
wrote:
Hi Frank,
Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as
it seems on Linux.
As you suggested when using the broker the slowdown after client disconnect disappears. However, the
broker seems to be very slow.
We were testing it with the same C# classes that showed acceptable performance without the broker. Using
the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.
Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available
cores evenly.
How is the performance on Linux?
We’re using
-
Windows 8
-
Broker, publisher and client on the same machine
-
Qpid 0.32 & Proton 0.7
Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without,
no communication happens.
Regards
Mathias
|
|
|
