Date   

Re: OpenMAMA-2.3.3-rc1

Alpert, Reed <reed.alpert@...>
 

Hi,

 

Is 2.3.3 being released soon?

We have some dependencies on that/etc.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Adrienne Ambrose [mailto:a.ambrose@...]
Sent: Wednesday, May 27, 2015 10:42 AM
To: Alpert, Reed; Gary Molloy; openmama-dev@...
Subject: RE: OpenMAMA-2.3.3-rc1

 

Hi Reed,

 

Thanks for the feedback, it is greatly appreciated.

Please keep us informed on the outcome of any further testing.

 

Thanks,

Adrienne

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Alpert, Reed
Sent: 27 May 2015 15:29
To: Gary Molloy; openmama-dev@...
Subject: Re: [Openmama-dev] OpenMAMA-2.3.3-rc1

 

We’ve been testing with this release and have no issues so far.

Testing C++ and Java clients.

Publish and subscribe.

Solace and Tick42 bridges.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Gary Molloy
Sent: Tuesday, May 05, 2015 10:31 AM
To: openmama-dev@...
Subject: [Openmama-dev] OpenMAMA-2.3.3-rc1

 

Hi Guys,

 

I have cut the new OpenMAMA-2.3.3 branch and created the OpenMAMA-2.3.3-rc1 tag, this is now available for testing.  I would anticipate a test period of around 2-3 weeks with a view to making the release official by the end of the month.

 

This release has been cut from the next branch.

 

The following list of issues/features have been added for this release:

 

BZ-166 Wombat: wInterlocked_set inconsistent return value

BZ-164 MAMAJNI: MamaPublisher: Overloaded MamaPublisher create method

BZ-169  Wombat queue has no separate deallocate method

BZ-176  Missing actions for snapshot subscriptions transition to deactivate state

BZ-168 Complete support for Vector Bool and Vector Char field types

BZ-156 No value expansion of last property line in mama.properties

BZ-178 Problem with mamaDictionary_getDictionaryMessage when multiple bridges are loaded

BZ-182 MAMAJAVA: Add java MamaDateTime::getAsFormattedString() method

BZ-181 MAMAJAVA: Java subscription setup fix - it loses the closure

BZ-183 SCons: OpenMAMA will not build on windows

BZ-189 [MAMAC] mamaPlugin Feature

BZ-190 [MAMAC] Add ability to turn on/off entitlements on a per bridge basis

BZ-191 [MAMAC] mamaPublisherImpl_getTransportImpl() accessor

BZ-179 OpenMAMA mock RPM's fail to build

BZ-187 SCons: Include stdout for build commands using site scons logger

BZ-188 Scons: fixes for Windows and Linux

BZ-192 OpenMAMA RPM Release Scripts

 

 

Thanks,

Gary

 

 

Gary Molloy – SR Labs

Adelaide Exchange | 24-26 Adelaide Street | Belfast | BT2 8GD

g.molloy@...

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Re: C# DQPublisher Manager

Mathias Kim
 

Hi Frank,

 

I tested the broker on a Fedora Virtualbox and the results were much better than on my Windows version. The performance was as good as with point to point for capturereplayc/mamalistenc (5 Symbols) and our C# classes.

For testing the C# classes I was simulating orderbooks. In the end I was sending an average of 14 messages per second. Each message contained on average 25 fields.

The same test with a Windows broker led to huge delay. Since the amount of data is fairly low, something must have gone wrong when compiling qpid under Windows.

Interestingly, also under Fedora I have to add the –t parameter. Without, no communication happens.

 

Unfortunately, with Virtualbox/Fedora our C# publisher and listener experience some problems since not every subscription is successful. In about 50% of the cases I get this behavior:

-          Publisher connects successfully to qpid broker

-          Listener connects successfully to qpid broker and submits desired symbol

-          Symbol arrives at publisher, publisher starts sending data to qpid broker

-          Qpid broker receives data (as I can see on the output) but never forwards to the listener

Monitoring the port shows that no messages ever arrive at the machine the listener is running on.

No similar behavior with capturereplayc/mamalistenc. Also, no similar issues with qpid under Windows.

 

Did you test the qpid broker/openmama combo with a C# Listener yet?

 

Regards

 

Mathias

 

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Montag, 22. Juni 2015 17:40
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

 

I was only testing with producer / consumer which is only one symbol so perhaps that's part of the issue here. I'll test with more symbols too when I get back to my machine. What sort of data rates were you running at?

 

Cheers,

Frank

 

On Mon, Jun 22, 2015 at 4:25 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

thanks for the answer.

 

I tried using Proton 0.8 but that changed nothing.

Also, I checked the queue depth. Publisher and Subscriber app had a depth of 0.

In addition, I checked the broker with capturereplayc and mamalistenc. More or less the same results: with one symbol there was no visible latency. After subscribing to 5 symbols I saw the same behavior as with our C# Publisher/Subscriber. So I think the issue lies with the Windows Broker. Also, I was running it on a Server within my network and the delay got even larger.

So I think the delay is caused by the (Windows) Qpid broker.

 

Since I have no clue how to modify the Windows Qpid broker, I’m going to test it with a Fedora Virtual Machine.

 

Regards,

 

Mathias

 

 

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Samstag, 20. Juni 2015 13:09
To: Mathias Kim
Cc:
openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

I just tried an equivalent test with Qpid Broker 0.32 and proton 0.9.1 (with a few small changes to let it compile as proton changed their interface again) and the producer keeps crashing... so we'll get back to what's going on there later.

So I tried with Proton 0.7 on linux (Fedora 22) and I can see:

1. Logging doesn't need to be disabled - seems to work regardless.

2. Performance is pretty much the same with the broker as it is point to point

3. CPU is spread across several cores

4. Latency is around 4ms at data rates of around 10,000 msg/s

If you're getting latencies in the 2 second region, my first guess is that you're queuing in your client, have you tried printing out the queue size in your application? This is all finger in the air stuff though because this is all running on my local laptop.

When you're operating point to point, the producer is essentially throttled by the speed of the consumer. When a broker is involved, this relationship is effectively decoupled, so you may find that the producer is actually publishing faster than you expect now (enqueuing for sending is not the same as actually sending). That or else your applications haven't been compiled with multithreading support and they're all just fighting with each other for scheduler time. I'll try a windows test when I get a chance to see if I can replicate what you're seeing.

What sort of data rates and message sizes were you putting through?

 

Cheers,

Frank

 

On Tue, Jun 16, 2015 at 4:31 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as it seems on Linux.

As you suggested when using the broker the slowdown after client disconnect disappears. However, the broker seems to be very slow.

 

We were testing it with the same C# classes that showed acceptable performance without the broker. Using the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.

Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available cores evenly.

 

How is the performance on Linux?

 

We’re using

-          Windows 8

-          Broker, publisher and client on the same machine

-          Qpid 0.32 & Proton 0.7

 

Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without, no communication happens.

 

Regards

 

Mathias

 

 


Re: NULL symbol causes crash

Gary Molloy <g.molloy@...>
 

Hi Reed,

 

Thanks for your email.

 

I must say, I have not heard of anyone using a NULL symbol to subscribe to the entire topic range for a source.  A symbol list subscription would be the method to use for that. 

I would be interested to hear if anyone in the community is using this method (NULL symbol to subscribe to an entire source).

 

But please do submit your patch J

 

Thanks,

Gary

 

Gary Molloy – SR Labs

Adelaide Exchange | 24-26 Adelaide Street | Belfast | BT2 8GD

g.molloy@...

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Alpert, Reed
Sent: 25 June 2015 16:01
To: openmama-dev@...
Subject: [Openmama-dev] NULL symbol causes crash

 

Hi,

 

Running modified mamalistencpp and using a NULL symbol causes a crash.

Granted that NULL symbols are not that easy to use, and there is some argument that says if someone does that it is best to remove their process from the market data ecosystem.

 

    mSource = "IDN_DEV";

    subscribeToSymbol(NULL);

 

Here is the trace, I’ll submit a patch, but wanted to know if there is any usage when a NULL symbol means the source has the entire topic.

 

(gdb) where

#0  0x00007fd30a31357f in __strlen_sse42 () from /lib64/libc.so.6

#1  0x00007fd30afcff6b in copyString (str=0x0) at mama/c_cpp/src/c/subscription.c:2432

#2  0x00007fd30afd1b51 in mamaSubscription_setupBasic (subscription=0x6dc930, transport=0x6b9350, queue=0x6db920, callbacks=0x7fd30af8fa40,

    source=0x40bfe2 "IDN_DEV", symbol=0x0, closure=0x6dcc40) at mama/c_cpp/src/c/subscription.c:465

#3  0x00007fd30afd209a in mamaSubscription_setup2 (subscription=0x6dc930, transport=0x6b9350, queue=0x6db920, callbacks=0x7fd30af8fa40,

    sourceName=<value optimized out>, symbol=<value optimized out>, closure=0x6dcc40) at mama/c_cpp/src/c/subscription.c:2868

#4  0x00007fd30ad6d340 in Wombat::MamaSubscription::setup (this=0x6dc8d0, transport=0x6bdc50, queue=0x6db8c0, callback=0x6dc8b0, source=0x40bfe2 "IDN_DEV",

    symbol=0x0, closure=0x0) at mama/c_cpp/src/cpp/MamaSubscription.cpp:525

#5  0x00007fd30ad6ca2d in Wombat::MamaSubscription::create (this=0x6dc8d0, transport=<value optimized out>, queue=<value optimized out>,

    callback=<value optimized out>, source=<value optimized out>, symbol=<value optimized out>, closure=0x0) at mama/c_cpp/src/cpp/MamaSubscription.cpp:443

#6  0x00000000004060e9 in MamaListen::subscribeToSymbol (this=0x7fffafc64740, symbol=0x0) at mamalistencpp.cpp:734

#7  0x0000000000406481 in MamaListen::subscribeToSymbols (this=0x7fffafc64740) at mamalistencpp.cpp:790

#8  0x0000000000408898 in main (argc=11, argv=0x7fffafc649a8) at mamalistencpp.cpp:1627

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


NULL symbol causes crash

Alpert, Reed <reed.alpert@...>
 

Hi,

 

Running modified mamalistencpp and using a NULL symbol causes a crash.

Granted that NULL symbols are not that easy to use, and there is some argument that says if someone does that it is best to remove their process from the market data ecosystem.

 

    mSource = "IDN_DEV";

    subscribeToSymbol(NULL);

 

Here is the trace, I’ll submit a patch, but wanted to know if there is any usage when a NULL symbol means the source has the entire topic.

 

(gdb) where

#0  0x00007fd30a31357f in __strlen_sse42 () from /lib64/libc.so.6

#1  0x00007fd30afcff6b in copyString (str=0x0) at mama/c_cpp/src/c/subscription.c:2432

#2  0x00007fd30afd1b51 in mamaSubscription_setupBasic (subscription=0x6dc930, transport=0x6b9350, queue=0x6db920, callbacks=0x7fd30af8fa40,

    source=0x40bfe2 "IDN_DEV", symbol=0x0, closure=0x6dcc40) at mama/c_cpp/src/c/subscription.c:465

#3  0x00007fd30afd209a in mamaSubscription_setup2 (subscription=0x6dc930, transport=0x6b9350, queue=0x6db920, callbacks=0x7fd30af8fa40,

    sourceName=<value optimized out>, symbol=<value optimized out>, closure=0x6dcc40) at mama/c_cpp/src/c/subscription.c:2868

#4  0x00007fd30ad6d340 in Wombat::MamaSubscription::setup (this=0x6dc8d0, transport=0x6bdc50, queue=0x6db8c0, callback=0x6dc8b0, source=0x40bfe2 "IDN_DEV",

    symbol=0x0, closure=0x0) at mama/c_cpp/src/cpp/MamaSubscription.cpp:525

#5  0x00007fd30ad6ca2d in Wombat::MamaSubscription::create (this=0x6dc8d0, transport=<value optimized out>, queue=<value optimized out>,

    callback=<value optimized out>, source=<value optimized out>, symbol=<value optimized out>, closure=0x0) at mama/c_cpp/src/cpp/MamaSubscription.cpp:443

#6  0x00000000004060e9 in MamaListen::subscribeToSymbol (this=0x7fffafc64740, symbol=0x0) at mamalistencpp.cpp:734

#7  0x0000000000406481 in MamaListen::subscribeToSymbols (this=0x7fffafc64740) at mamalistencpp.cpp:790

#8  0x0000000000408898 in main (argc=11, argv=0x7fffafc649a8) at mamalistencpp.cpp:1627

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Re: C# DQPublisher Manager

Frank Quinn <fquinn.ni@...>
 

Hi Mathias,

I was only testing with producer / consumer which is only one symbol so perhaps that's part of the issue here. I'll test with more symbols too when I get back to my machine. What sort of data rates were you running at?

Cheers,
Frank

On Mon, Jun 22, 2015 at 4:25 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

thanks for the answer.

 

I tried using Proton 0.8 but that changed nothing.

Also, I checked the queue depth. Publisher and Subscriber app had a depth of 0.

In addition, I checked the broker with capturereplayc and mamalistenc. More or less the same results: with one symbol there was no visible latency. After subscribing to 5 symbols I saw the same behavior as with our C# Publisher/Subscriber. So I think the issue lies with the Windows Broker. Also, I was running it on a Server within my network and the delay got even larger.

So I think the delay is caused by the (Windows) Qpid broker.

 

Since I have no clue how to modify the Windows Qpid broker, I’m going to test it with a Fedora Virtual Machine.

 

Regards,

 

Mathias

 

 

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Samstag, 20. Juni 2015 13:09
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

I just tried an equivalent test with Qpid Broker 0.32 and proton 0.9.1 (with a few small changes to let it compile as proton changed their interface again) and the producer keeps crashing... so we'll get back to what's going on there later.

So I tried with Proton 0.7 on linux (Fedora 22) and I can see:

1. Logging doesn't need to be disabled - seems to work regardless.

2. Performance is pretty much the same with the broker as it is point to point

3. CPU is spread across several cores

4. Latency is around 4ms at data rates of around 10,000 msg/s

If you're getting latencies in the 2 second region, my first guess is that you're queuing in your client, have you tried printing out the queue size in your application? This is all finger in the air stuff though because this is all running on my local laptop.

When you're operating point to point, the producer is essentially throttled by the speed of the consumer. When a broker is involved, this relationship is effectively decoupled, so you may find that the producer is actually publishing faster than you expect now (enqueuing for sending is not the same as actually sending). That or else your applications haven't been compiled with multithreading support and they're all just fighting with each other for scheduler time. I'll try a windows test when I get a chance to see if I can replicate what you're seeing.

What sort of data rates and message sizes were you putting through?

 

Cheers,

Frank

 

On Tue, Jun 16, 2015 at 4:31 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as it seems on Linux.

As you suggested when using the broker the slowdown after client disconnect disappears. However, the broker seems to be very slow.

 

We were testing it with the same C# classes that showed acceptable performance without the broker. Using the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.

Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available cores evenly.

 

How is the performance on Linux?

 

We’re using

-          Windows 8

-          Broker, publisher and client on the same machine

-          Qpid 0.32 & Proton 0.7

 

Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without, no communication happens.

 

Regards

 

Mathias

 



Re: C# DQPublisher Manager

Mathias Kim
 

Hi Frank,

 

thanks for the answer.

 

I tried using Proton 0.8 but that changed nothing.

Also, I checked the queue depth. Publisher and Subscriber app had a depth of 0.

In addition, I checked the broker with capturereplayc and mamalistenc. More or less the same results: with one symbol there was no visible latency. After subscribing to 5 symbols I saw the same behavior as with our C# Publisher/Subscriber. So I think the issue lies with the Windows Broker. Also, I was running it on a Server within my network and the delay got even larger.

So I think the delay is caused by the (Windows) Qpid broker.

 

Since I have no clue how to modify the Windows Qpid broker, I’m going to test it with a Fedora Virtual Machine.

 

Regards,

 

Mathias

 

 

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Samstag, 20. Juni 2015 13:09
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

I just tried an equivalent test with Qpid Broker 0.32 and proton 0.9.1 (with a few small changes to let it compile as proton changed their interface again) and the producer keeps crashing... so we'll get back to what's going on there later.

So I tried with Proton 0.7 on linux (Fedora 22) and I can see:

1. Logging doesn't need to be disabled - seems to work regardless.

2. Performance is pretty much the same with the broker as it is point to point

3. CPU is spread across several cores

4. Latency is around 4ms at data rates of around 10,000 msg/s

If you're getting latencies in the 2 second region, my first guess is that you're queuing in your client, have you tried printing out the queue size in your application? This is all finger in the air stuff though because this is all running on my local laptop.

When you're operating point to point, the producer is essentially throttled by the speed of the consumer. When a broker is involved, this relationship is effectively decoupled, so you may find that the producer is actually publishing faster than you expect now (enqueuing for sending is not the same as actually sending). That or else your applications haven't been compiled with multithreading support and they're all just fighting with each other for scheduler time. I'll try a windows test when I get a chance to see if I can replicate what you're seeing.

What sort of data rates and message sizes were you putting through?

 

Cheers,

Frank

 

On Tue, Jun 16, 2015 at 4:31 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as it seems on Linux.

As you suggested when using the broker the slowdown after client disconnect disappears. However, the broker seems to be very slow.

 

We were testing it with the same C# classes that showed acceptable performance without the broker. Using the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.

Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available cores evenly.

 

How is the performance on Linux?

 

We’re using

-          Windows 8

-          Broker, publisher and client on the same machine

-          Qpid 0.32 & Proton 0.7

 

Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without, no communication happens.

 

Regards

 

Mathias

 


Re: C# DQPublisher Manager

Frank Quinn <fquinn.ni@...>
 

Hi Mathias,

I just tried an equivalent test with Qpid Broker 0.32 and proton 0.9.1 (with a few small changes to let it compile as proton changed their interface again) and the producer keeps crashing... so we'll get back to what's going on there later.

So I tried with Proton 0.7 on linux (Fedora 22) and I can see:

1. Logging doesn't need to be disabled - seems to work regardless.
2. Performance is pretty much the same with the broker as it is point to point
3. CPU is spread across several cores
4. Latency is around 4ms at data rates of around 10,000 msg/s

If you're getting latencies in the 2 second region, my first guess is that you're queuing in your client, have you tried printing out the queue size in your application? This is all finger in the air stuff though because this is all running on my local laptop.

When you're operating point to point, the producer is essentially throttled by the speed of the consumer. When a broker is involved, this relationship is effectively decoupled, so you may find that the producer is actually publishing faster than you expect now (enqueuing for sending is not the same as actually sending). That or else your applications haven't been compiled with multithreading support and they're all just fighting with each other for scheduler time. I'll try a windows test when I get a chance to see if I can replicate what you're seeing.

What sort of data rates and message sizes were you putting through?

Cheers,
Frank

On Tue, Jun 16, 2015 at 4:31 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as it seems on Linux.

As you suggested when using the broker the slowdown after client disconnect disappears. However, the broker seems to be very slow.

 

We were testing it with the same C# classes that showed acceptable performance without the broker. Using the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.

Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available cores evenly.

 

How is the performance on Linux?

 

We’re using

-          Windows 8

-          Broker, publisher and client on the same machine

-          Qpid 0.32 & Proton 0.7

 

Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without, no communication happens.

 

Regards

 

Mathias



Re: C# DQPublisher Manager

Mathias Kim
 

Hi Frank,

 

Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as it seems on Linux.

As you suggested when using the broker the slowdown after client disconnect disappears. However, the broker seems to be very slow.

 

We were testing it with the same C# classes that showed acceptable performance without the broker. Using the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.

Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available cores evenly.

 

How is the performance on Linux?

 

We’re using

-          Windows 8

-          Broker, publisher and client on the same machine

-          Qpid 0.32 & Proton 0.7

 

Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without, no communication happens.

 

Regards

 

Mathias


Tick42 bidge for TREP

Tom Doust
 

Hi,

 

Many of you will be aware that Tick42 have been developing an OpenMAMA bridge for Thomson Reuters TREP.  You may also have noticed that it was publicly announced yesterday.

 

There is a press release at http://www.finextra.com/news/announcement.aspx?pressreleaseid=60079  and details of how to get it at http://tick42.com/trepbridge

 

The Tick42 Bridge provides support for subscribing to L1 and L2 data (full book or aggregated by price) and published amam messages as either an interactive or non-interactive provider. It supports contribution by posting rssl messages. It includes flexible mapping between TR and mama/mamda fieldnames, field ids and types. In addition to the open source release we have an enhanced version that is available under licence from Tick42 that includes source discovery, the ability to accept posted messages, handling legacy MFEED messages, and, in the next release, the ability to handle ANSI page data.

 

Please note that you will need a licenced UPA SDK from Thomson Reuters in order to build and run the code and obviously a TREP infrastructure to connect to.

 

Please feel free to contact us either directly or through the OpenMAMA mailing lists if you have questions or comments.

 

Best Regards

 

Tom Doust

 

 

TOM DOUST | Head of Consultancy                                                                                                         


TICK42

P: +44 (0) 1628 477444 | M: +44 (0) 7710 479924 | E: tom.doust@... | skype:  tom.doust |  http://www.tick42.com

 

 


Re: Bloomberg and OpenMAMA

Sanjeev Wahi <wahi@...>
 

Hi Nestor, I did something similar (Making a Proto-Type of arca-order-book/algo-trading & QuantLib www.quantlib.org ) for a retail client in the past (using MAMDA/Wombat when it was owned by NYSE). (I did not use Tick42, so I cannot speak for Tick42 but with Wombat it works like following)

 

I developed few generic-utility templates using C/C++ OpenMAMA/OpenMAMDA API.

 

The data-flow works like following:

 

 

 

 

 

 

For details refer page # 22 of the attached OpenMAMDA doc.

 

Hope that helps,

 

-Best Regards,

Sanjeev

 

 

 

 

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Macrux
Sent: Thursday, June 11, 2015 12:53 PM
To: Tom Doust
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] Bloomberg and OpenMAMA

 

A link to the image, if you can't see it: image

 

On 11 June 2015 at 11:48, Macrux <kmacrux@...> wrote:

Hi Gary, Tom,

I'm going to clarify the scenario:

I'm trying to build an Automatic Trading System on the OpenMAMA basis. The initial market data provider is Bloomberg (a normal subscription, not b-pipe). I want to run different algorithms where each one is a separated instance, like the figure shows:




Now, thinking about the Tom answer, right now it seems not to be possible to get the order book using the tick42 bridge, do you know how could I accomplish this requirement?

Do you have any suggestion about a better way to build this algorithmic trading system?

Thanks in advance and,

Kind regards,

 

On 10 June 2015 at 11:21, Tom Doust <tom.doust@...> wrote:

Hi Nestor

 

Just to add to Gary’s reply. The Bloomberg API will support multiple connections on the same source. The number of different symbols you can subscribe to is limited only by what you are entitled to and by any subscription limits that Bloomberg may apply to your account.

 

Currently, the Tick42 blp  bridge does not generate mamda book messages. This is planned for a future version, but not yet scheduled

 

I hope this answers you questions, but please feel free to contact us if you need any further information.

 

Best regards

 

Tom Doust

 

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Gary Molloy
Sent: 08 June 2015 15:37
To: Macrux; openmama-dev@...
Subject: Re: [Openmama-dev] Bloomberg and OpenMAMA

 

Hi Nestor,

 

Thanks for your email.

 

Could I ask you to clarify your scenario a bit more please?  For example when you say you have multiple subscribing applications, would these be separate instances of the application, or perhaps with the same process?  From an OpenMAMA perspective there shouldn’t be any problem with making multiple subscriptions to the same source via the same bridge.

 

For your second query, yes OpenMAMDA is capable handling orderbooks.

 

Thanks,

Gary

 

 

Gary Molloy – SR Labs

Adelaide Exchange | 24-26 Adelaide Street | Belfast | BT2 8GD

g.molloy@...

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Macrux
Sent: 28 May 2015 18:34
To: openmama-dev@...
Subject: [Openmama-dev] Bloomberg and OpenMAMA

 

Hi there,

I have some doubts about OpenMAMA and Bloomberg. In first place, I'm wondering if it is possible to conect multiple subscriber applications to bloomberg (using the tick42blp middleware bridge), each one subscribed to the same source (the name space for the data, e.g. "BlpMktData"), but subscribed to a different symbols, something  like this:
 ___________     ___________    ___________

| subscriber-1 |    | subscriber-2 |  | subscriber-n |           
|___________|     |___________|  |___________|
_______|____________|_____________|_______ 

|                         openMAMA                            |

|________________________________________|
                    _______|________

                   | bloomberg bridge  |
                   |________________|

*subscriber-1, 2 and n are subscribed to diferent individual symbols.

In second place, is it possible to use openMAMDA to get the order books for each one of those subscribed symbols with the current available bridge from Tick42?

And that's all. Thank you for your help, I will really appreciate it, and other additional information or advices you want to give me.

Best regards,

Nestor.

 

 

 

 


Re: Bloomberg and OpenMAMA

macrux
 

A link to the image, if you can't see it: image

On 11 June 2015 at 11:48, Macrux <kmacrux@...> wrote:
Hi Gary, Tom,

I'm going to clarify the scenario:

I'm trying to build an Automatic Trading System on the OpenMAMA basis. The initial market data provider is Bloomberg (a normal subscription, not b-pipe). I want to run different algorithms where each one is a separated instance, like the figure shows:





Now, thinking about the Tom answer, right now it seems not to be possible to get the order book using the tick42 bridge, do you know how could I accomplish this requirement?

Do you have any suggestion about a better way to build this algorithmic trading system?

Thanks in advance and,

Kind regards,

On 10 June 2015 at 11:21, Tom Doust <tom.doust@...> wrote:

Hi Nestor

 

Just to add to Gary’s reply. The Bloomberg API will support multiple connections on the same source. The number of different symbols you can subscribe to is limited only by what you are entitled to and by any subscription limits that Bloomberg may apply to your account.

 

Currently, the Tick42 blp  bridge does not generate mamda book messages. This is planned for a future version, but not yet scheduled

 

I hope this answers you questions, but please feel free to contact us if you need any further information.

 

Best regards

 

Tom Doust

 

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Gary Molloy
Sent: 08 June 2015 15:37
To: Macrux; openmama-dev@...
Subject: Re: [Openmama-dev] Bloomberg and OpenMAMA

 

Hi Nestor,

 

Thanks for your email.

 

Could I ask you to clarify your scenario a bit more please?  For example when you say you have multiple subscribing applications, would these be separate instances of the application, or perhaps with the same process?  From an OpenMAMA perspective there shouldn’t be any problem with making multiple subscriptions to the same source via the same bridge.

 

For your second query, yes OpenMAMDA is capable handling orderbooks.

 

Thanks,

Gary

 

 

Gary Molloy – SR Labs

Adelaide Exchange | 24-26 Adelaide Street | Belfast | BT2 8GD

g.molloy@...

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Macrux
Sent: 28 May 2015 18:34
To: openmama-dev@...
Subject: [Openmama-dev] Bloomberg and OpenMAMA

 

Hi there,

I have some doubts about OpenMAMA and Bloomberg. In first place, I'm wondering if it is possible to conect multiple subscriber applications to bloomberg (using the tick42blp middleware bridge), each one subscribed to the same source (the name space for the data, e.g. "BlpMktData"), but subscribed to a different symbols, something  like this:
 ___________     ___________    ___________

| subscriber-1 |    | subscriber-2 |  | subscriber-n |           
|___________|     |___________|  |___________|
_______|____________|_____________|_______ 

|                         openMAMA                            |

|________________________________________|
                    _______|________

                   | bloomberg bridge  |
                   |________________|

*subscriber-1, 2 and n are subscribed to diferent individual symbols.

In second place, is it possible to use openMAMDA to get the order books for each one of those subscribed symbols with the current available bridge from Tick42?

And that's all. Thank you for your help, I will really appreciate it, and other additional information or advices you want to give me.

Best regards,

Nestor.

 






Re: Bloomberg and OpenMAMA

macrux
 

Hi Gary, Tom,

I'm going to clarify the scenario:

I'm trying to build an Automatic Trading System on the OpenMAMA basis. The initial market data provider is Bloomberg (a normal subscription, not b-pipe). I want to run different algorithms where each one is a separated instance, like the figure shows:





Now, thinking about the Tom answer, right now it seems not to be possible to get the order book using the tick42 bridge, do you know how could I accomplish this requirement?

Do you have any suggestion about a better way to build this algorithmic trading system?

Thanks in advance and,

Kind regards,

On 10 June 2015 at 11:21, Tom Doust <tom.doust@...> wrote:

Hi Nestor

 

Just to add to Gary’s reply. The Bloomberg API will support multiple connections on the same source. The number of different symbols you can subscribe to is limited only by what you are entitled to and by any subscription limits that Bloomberg may apply to your account.

 

Currently, the Tick42 blp  bridge does not generate mamda book messages. This is planned for a future version, but not yet scheduled

 

I hope this answers you questions, but please feel free to contact us if you need any further information.

 

Best regards

 

Tom Doust

 

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Gary Molloy
Sent: 08 June 2015 15:37
To: Macrux; openmama-dev@...
Subject: Re: [Openmama-dev] Bloomberg and OpenMAMA

 

Hi Nestor,

 

Thanks for your email.

 

Could I ask you to clarify your scenario a bit more please?  For example when you say you have multiple subscribing applications, would these be separate instances of the application, or perhaps with the same process?  From an OpenMAMA perspective there shouldn’t be any problem with making multiple subscriptions to the same source via the same bridge.

 

For your second query, yes OpenMAMDA is capable handling orderbooks.

 

Thanks,

Gary

 

 

Gary Molloy – SR Labs

Adelaide Exchange | 24-26 Adelaide Street | Belfast | BT2 8GD

g.molloy@...

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Macrux
Sent: 28 May 2015 18:34
To: openmama-dev@...
Subject: [Openmama-dev] Bloomberg and OpenMAMA

 

Hi there,

I have some doubts about OpenMAMA and Bloomberg. In first place, I'm wondering if it is possible to conect multiple subscriber applications to bloomberg (using the tick42blp middleware bridge), each one subscribed to the same source (the name space for the data, e.g. "BlpMktData"), but subscribed to a different symbols, something  like this:
 ___________     ___________    ___________

| subscriber-1 |    | subscriber-2 |  | subscriber-n |           
|___________|     |___________|  |___________|
_______|____________|_____________|_______ 

|                         openMAMA                            |

|________________________________________|
                    _______|________

                   | bloomberg bridge  |
                   |________________|

*subscriber-1, 2 and n are subscribed to diferent individual symbols.

In second place, is it possible to use openMAMDA to get the order books for each one of those subscribed symbols with the current available bridge from Tick42?

And that's all. Thank you for your help, I will really appreciate it, and other additional information or advices you want to give me.

Best regards,

Nestor.

 





Re: Bloomberg and OpenMAMA

Tom Doust
 

Hi Nestor

 

Just to add to Gary’s reply. The Bloomberg API will support multiple connections on the same source. The number of different symbols you can subscribe to is limited only by what you are entitled to and by any subscription limits that Bloomberg may apply to your account.

 

Currently, the Tick42 blp  bridge does not generate mamda book messages. This is planned for a future version, but not yet scheduled

 

I hope this answers you questions, but please feel free to contact us if you need any further information.

 

Best regards

 

Tom Doust

 

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Gary Molloy
Sent: 08 June 2015 15:37
To: Macrux; openmama-dev@...
Subject: Re: [Openmama-dev] Bloomberg and OpenMAMA

 

Hi Nestor,

 

Thanks for your email.

 

Could I ask you to clarify your scenario a bit more please?  For example when you say you have multiple subscribing applications, would these be separate instances of the application, or perhaps with the same process?  From an OpenMAMA perspective there shouldn’t be any problem with making multiple subscriptions to the same source via the same bridge.

 

For your second query, yes OpenMAMDA is capable handling orderbooks.

 

Thanks,

Gary

 

 

Gary Molloy – SR Labs

Adelaide Exchange | 24-26 Adelaide Street | Belfast | BT2 8GD

g.molloy@...

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Macrux
Sent: 28 May 2015 18:34
To: openmama-dev@...
Subject: [Openmama-dev] Bloomberg and OpenMAMA

 

Hi there,

I have some doubts about OpenMAMA and Bloomberg. In first place, I'm wondering if it is possible to conect multiple subscriber applications to bloomberg (using the tick42blp middleware bridge), each one subscribed to the same source (the name space for the data, e.g. "BlpMktData"), but subscribed to a different symbols, something  like this:
 ___________     ___________    ___________

| subscriber-1 |    | subscriber-2 |  | subscriber-n |           
|___________|     |___________|  |___________|
_______|____________|_____________|_______ 

|                         openMAMA                            |

|________________________________________|
                    _______|________

                   | bloomberg bridge  |
                   |________________|

*subscriber-1, 2 and n are subscribed to diferent individual symbols.

In second place, is it possible to use openMAMDA to get the order books for each one of those subscribed symbols with the current available bridge from Tick42?

And that's all. Thank you for your help, I will really appreciate it, and other additional information or advices you want to give me.

Best regards,

Nestor.

 




Re: Bloomberg and OpenMAMA

Gary Molloy <g.molloy@...>
 

Hi Nestor,

 

Thanks for your email.

 

Could I ask you to clarify your scenario a bit more please?  For example when you say you have multiple subscribing applications, would these be separate instances of the application, or perhaps with the same process?  From an OpenMAMA perspective there shouldn’t be any problem with making multiple subscriptions to the same source via the same bridge.

 

For your second query, yes OpenMAMDA is capable handling orderbooks.

 

Thanks,

Gary

 

 

Gary Molloy – SR Labs

Adelaide Exchange | 24-26 Adelaide Street | Belfast | BT2 8GD

g.molloy@...

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Macrux
Sent: 28 May 2015 18:34
To: openmama-dev@...
Subject: [Openmama-dev] Bloomberg and OpenMAMA

 

Hi there,

I have some doubts about OpenMAMA and Bloomberg. In first place, I'm wondering if it is possible to conect multiple subscriber applications to bloomberg (using the tick42blp middleware bridge), each one subscribed to the same source (the name space for the data, e.g. "BlpMktData"), but subscribed to a different symbols, something  like this:
 ___________     ___________    ___________

| subscriber-1 |    | subscriber-2 |  | subscriber-n |           
|___________|     |___________|  |___________|
_______|____________|_____________|_______ 

|                         openMAMA                            |

|________________________________________|
                    _______|________

                   | bloomberg bridge  |
                   |________________|

*subscriber-1, 2 and n are subscribed to diferent individual symbols.

In second place, is it possible to use openMAMDA to get the order books for each one of those subscribed symbols with the current available bridge from Tick42?

And that's all. Thank you for your help, I will really appreciate it, and other additional information or advices you want to give me.

Best regards,

Nestor.

 





Re: C# DQPublisher Manager

Frank Quinn <fquinn.ni@...>
 

Oh - and with the broker implementation, you no longer require a separate transport for pub and sub unless they're on different subnets - you just point both sides to the broker.

Cheers,
Frank

On Fri, Jun 5, 2015 at 1:32 PM, Frank Quinn <fquinn.ni@...> wrote:
Hi Mathias,

The best documentation for this was probably on the original commit note:


It contains an example transport - all you need to to is change your IPs to the IP address of qpidd (running as described) and then just run your MAMA example applications using -m qpid -tport qpidbroker.

Note it contains a new mama.qpid.transport.<transportname>.type parameter to let the bridge know that this is a broker transport.

Cheers,
Frank

On Fri, Jun 5, 2015 at 1:20 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

thanks for looking into this.

We’re now having a look into the qpid bridge with broker. We compiled the bridge but then got stuck using it.

Can you provide documentation on how to use the broker feature with the examples, e.g. capturereplayc?

 

Regards

 

Mathias

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Freitag, 5. Juni 2015 00:39
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

 

Tracking this via http://bugs.openmama.org/show_bug.cgi?id=41 – looks like the proton folks added something in 0.8 to allow you to pull out link information from a messenger which we may be able to take advantage of. It looks like it’ll involve a little expansion of the endpoint code to store more information about an endpoint than just its URI but it should be doable.

 

Cheers,

Frank

 

On Mon, Jun 1, 2015 at 7:03 PM, Frank Quinn <fquinn.ni@...> wrote:

Hi Mathias,

I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure.

In fact I just checked and we raised this with the proton guys 2 years ago and they never responded so maybe we should chase that up :) http://qpid.2158936.n2.nabble.com/Detecting-and-capturing-underlying-problems-and-event-with-pn-messenger-td7595635.html

To answer your more specific questions:

 - This is 100% qpid proton / MAMA - not your application's fault

 - The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this

 - This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will always be a little more interactive by nature

On a side note, have you tried our test Qpid Proton Broker implementation http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.

 

Cheers,

Frank

 

On Mon, Jun 1, 2015 at 4:00 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi guys,

we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
-       Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
-       Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
-       We initially thought that openmama/qpid just sends out (broadcasts) messages  and doesn't care whether they arrive or not. Is this the case?

Regards

Mathias


-----Original Message-----
From: Ian Bell [mailto:ian.bell@...]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@...
Subject: RE: [Openmama-dev] C# DQPublisher Manager

Hi

Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback.  As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk


-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@...]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager

Hi guys,

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?

Hope you can help us.
Best regards

Mathias Kim

ETF Analyst

Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>

Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner



The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>

_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

 




Re: C# DQPublisher Manager

Frank Quinn <fquinn.ni@...>
 

Hi Mathias,

The best documentation for this was probably on the original commit note:


It contains an example transport - all you need to to is change your IPs to the IP address of qpidd (running as described) and then just run your MAMA example applications using -m qpid -tport qpidbroker.

Note it contains a new mama.qpid.transport.<transportname>.type parameter to let the bridge know that this is a broker transport.

Cheers,
Frank

On Fri, Jun 5, 2015 at 1:20 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

thanks for looking into this.

We’re now having a look into the qpid bridge with broker. We compiled the bridge but then got stuck using it.

Can you provide documentation on how to use the broker feature with the examples, e.g. capturereplayc?

 

Regards

 

Mathias

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Freitag, 5. Juni 2015 00:39
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

 

Tracking this via http://bugs.openmama.org/show_bug.cgi?id=41 – looks like the proton folks added something in 0.8 to allow you to pull out link information from a messenger which we may be able to take advantage of. It looks like it’ll involve a little expansion of the endpoint code to store more information about an endpoint than just its URI but it should be doable.

 

Cheers,

Frank

 

On Mon, Jun 1, 2015 at 7:03 PM, Frank Quinn <fquinn.ni@...> wrote:

Hi Mathias,

I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure.

In fact I just checked and we raised this with the proton guys 2 years ago and they never responded so maybe we should chase that up :) http://qpid.2158936.n2.nabble.com/Detecting-and-capturing-underlying-problems-and-event-with-pn-messenger-td7595635.html

To answer your more specific questions:

 - This is 100% qpid proton / MAMA - not your application's fault

 - The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this

 - This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will always be a little more interactive by nature

On a side note, have you tried our test Qpid Proton Broker implementation http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.

 

Cheers,

Frank

 

On Mon, Jun 1, 2015 at 4:00 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi guys,

we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
-       Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
-       Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
-       We initially thought that openmama/qpid just sends out (broadcasts) messages  and doesn't care whether they arrive or not. Is this the case?

Regards

Mathias


-----Original Message-----
From: Ian Bell [mailto:ian.bell@...]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@...
Subject: RE: [Openmama-dev] C# DQPublisher Manager

Hi

Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback.  As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk


-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@...]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager

Hi guys,

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?

Hope you can help us.
Best regards

Mathias Kim

ETF Analyst

Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>

Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner



The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>

_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

 



Re: C# DQPublisher Manager

Mathias Kim
 

Hi Frank,

 

thanks for looking into this.

We’re now having a look into the qpid bridge with broker. We compiled the bridge but then got stuck using it.

Can you provide documentation on how to use the broker feature with the examples, e.g. capturereplayc?

 

Regards

 

Mathias

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Freitag, 5. Juni 2015 00:39
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

 

Tracking this via http://bugs.openmama.org/show_bug.cgi?id=41 – looks like the proton folks added something in 0.8 to allow you to pull out link information from a messenger which we may be able to take advantage of. It looks like it’ll involve a little expansion of the endpoint code to store more information about an endpoint than just its URI but it should be doable.

 

Cheers,

Frank

 

On Mon, Jun 1, 2015 at 7:03 PM, Frank Quinn <fquinn.ni@...> wrote:

Hi Mathias,

I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure.

In fact I just checked and we raised this with the proton guys 2 years ago and they never responded so maybe we should chase that up :) http://qpid.2158936.n2.nabble.com/Detecting-and-capturing-underlying-problems-and-event-with-pn-messenger-td7595635.html

To answer your more specific questions:

 - This is 100% qpid proton / MAMA - not your application's fault

 - The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this

 - This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will always be a little more interactive by nature

On a side note, have you tried our test Qpid Proton Broker implementation http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.

 

Cheers,

Frank

 

On Mon, Jun 1, 2015 at 4:00 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi guys,

we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
-       Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
-       Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
-       We initially thought that openmama/qpid just sends out (broadcasts) messages  and doesn't care whether they arrive or not. Is this the case?

Regards

Mathias


-----Original Message-----
From: Ian Bell [mailto:ian.bell@...]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@...
Subject: RE: [Openmama-dev] C# DQPublisher Manager

Hi

Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback.  As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk


-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@...]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager

Hi guys,

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?

Hope you can help us.
Best regards

Mathias Kim

ETF Analyst

Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>

Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner



The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>

_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

 


Re: C# DQPublisher Manager

Frank Quinn <fquinn.ni@...>
 

Hi Mathias,

 

Tracking this via http://bugs.openmama.org/show_bug.cgi?id=41 – looks like the proton folks added something in 0.8 to allow you to pull out link information from a messenger which we may be able to take advantage of. It looks like it’ll involve a little expansion of the endpoint code to store more information about an endpoint than just its URI but it should be doable.

 

Cheers,

Frank


On Mon, Jun 1, 2015 at 7:03 PM, Frank Quinn <fquinn.ni@...> wrote:
Hi Mathias,

I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure.

In fact I just checked and we raised this with the proton guys 2 years ago and they never responded so maybe we should chase that up :) http://qpid.2158936.n2.nabble.com/Detecting-and-capturing-underlying-problems-and-event-with-pn-messenger-td7595635.html

To answer your more specific questions:
 - This is 100% qpid proton / MAMA - not your application's fault
 - The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this
 - This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will always be a little more interactive by nature

On a side note, have you tried our test Qpid Proton Broker implementation http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.

Cheers,
Frank

On Mon, Jun 1, 2015 at 4:00 PM, Mathias Kim <Mathias.Kim@...> wrote:
Hi guys,

we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
-       Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
-       Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
-       We initially thought that openmama/qpid just sends out (broadcasts) messages  and doesn't care whether they arrive or not. Is this the case?

Regards

Mathias

-----Original Message-----
From: Ian Bell [mailto:ian.bell@...]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@...
Subject: RE: [Openmama-dev] C# DQPublisher Manager

Hi

Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback.  As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk


-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@...]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager

Hi guys,

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?

Hope you can help us.
Best regards

Mathias Kim

ETF Analyst

Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>

Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner



The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>

_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev



Re: C# DQPublisher Manager

Frank Quinn <fquinn.ni@...>
 

Hi Mathias,

I just tried a scenario similar to what you described with mamapublisherc and mamasubscriberc and I can confirm that I can see the same issue. I'll have a look at what Qpid Proton have to offer in their latest releases to handle this scenario but I remember looking at this before and the problem revolves around the fact that most of the time, our calls to pn_messenger_put and pn_messenger_send don't actually return any error codes in the event of failure. If you look in the code, you'll see that we actually do error checking on those function calls but they don't return any failure.

In fact I just checked and we raised this with the proton guys 2 years ago and they never responded so maybe we should chase that up :) http://qpid.2158936.n2.nabble.com/Detecting-and-capturing-underlying-problems-and-event-with-pn-messenger-td7595635.html

To answer your more specific questions:
 - This is 100% qpid proton / MAMA - not your application's fault
 - The bridge doesn't even know, so we can't tell the application until we get a good answer from the qpid proton folks or find some new mechanism that has been added to handle this
 - This is certainly the case with MAMA but the underlying middleware implementation may not be so flippant as is the case here - all connections in AMQP implementations are connection oriented so they will always be a little more interactive by nature

On a side note, have you tried our test Qpid Proton Broker implementation http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/feature-qpid-broker ? I expect it will make a lot of these sorts of problems go away and it will be a lot easier to configure (you don't need to specify a different port for each client, for example). Note that to use that branch, both clients and publishers will need to be compiled against it as the bridge's protocol messages have changed in that version.

Cheers,
Frank

On Mon, Jun 1, 2015 at 4:00 PM, Mathias Kim <Mathias.Kim@...> wrote:
Hi guys,

we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
-       Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
-       Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
-       We initially thought that openmama/qpid just sends out (broadcasts) messages  and doesn't care whether they arrive or not. Is this the case?

Regards

Mathias

-----Original Message-----
From: Ian Bell [mailto:ian.bell@...]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@...
Subject: RE: [Openmama-dev] C# DQPublisher Manager

Hi

Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback.  As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk


-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@...]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager

Hi guys,

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?

Hope you can help us.
Best regards

Mathias Kim

ETF Analyst

Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>

Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner



The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>

_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev


Re: C# DQPublisher Manager

Mathias Kim
 

Hi guys,

we managed to implement the dqpublishermanager and the dqpublisher in C# and most things seem to work well.
Setting up a server app with the qpid bridge works, as well as sending dummy market data to a client. Also, multiple clients can subscribe to the server and all get the messages reasonably fast.
Nevertheless, one huge issue remains:
Whenever one client disconnects, the server continues to send that symbol to that client. So far, this behavior was expected. Unfortunately, as soon as the client destroys its transport, sending takes a lot longer than before. Also, we get an error which depends on the qpid version (SASL header mismatch with v0.8, connection aborted with v0.7). This slows down the whole server until the client connects again. Then the server sends all the messages which it tried so send during the client's off time. As soon this is done, the server is reacting normal and works as before.
We now ask ourselves:
- Is this problem caused by our implementation of the dqpublisher/manager or is this rather a qpid issue?
- Is there any way for the server to get notified when a client disconnects/doesn't respond anymore?
- We initially thought that openmama/qpid just sends out (broadcasts) messages and doesn't care whether they arrive or not. Is this the case?

Regards

Mathias

-----Original Message-----
From: Ian Bell [mailto:ian.bell@hapsoftconsulting.co.uk]
Sent: Dienstag, 14. April 2015 13:49
To: Mathias Kim
Cc: openmama-dev@lists.openmama.org
Subject: RE: [Openmama-dev] C# DQPublisher Manager

Hi

Just wondering if you go this sorted?
The first request should route through to the OnNewRequst callback. As a guess I suspect that the topic is 100% correct, or possibly the callback delegates are not hooked up completely.
Can you supply the code and some logs and I may be able to spot the issue.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk


-----Original Message-----
From: Mathias Kim [mailto:Mathias.Kim@crossflow.de]
Sent: 25 March 2015 16:40
Subject: [Openmama-dev] C# DQPublisher Manager

Hi guys,

we continued wrapping the dqpublishermanager class in C# but got stuck again. We can create a dqpublishermanager and I also get the onCreate callback. Now, we try to notify the dqpm that there is interest in a certain symbol.
We try to connect to the dqpm via a qpid bridge. Unfortunately, the request never arrives at the dqpm so the OnNewRequest callback never gets called. However, the transport class recognizes the request and notifies us on the console. It even remembers that we previously shows interest in the same symbol. Therefore we like to know whether our implementation is faulty or whether we need to route the request from the transport to the dqpm ourselves.
How does this generally work?

Hope you can help us.
Best regards

Mathias Kim

ETF Analyst

Phone +49 89 442327-191
Mobile +49 171 4152665
mathias.kim at crossflow.de<mailto:mathias.kim at crossflow.de> www.crossflow.de<https://exchange.assenagon.com/owa/redir.aspx?C=tVJgRBMA7UG6CkQ3TBnr2S2utVWiDdIIBlqZxpqCNcW6tUTfy7PQJ8R4pX_VH_LEFJ8GJHinPD8.&URL=http%3a%2f%2fwww.crossflow.de>

Crossflow Financial Advisors GmbH
Sonnenstra?e 19
D-80331 M?nchen
Local Court of Munich HRB 186408
Managing Directors: Markus Deffner, Juergen Fritzen, Erol Steiner



The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this e-mail may be unlawful. If you have received this e-mail in error, please notify the sender immediately and delete it from your system. E-mail transmission cannot be guaranteed to be secure or error free, as messages can be intercepted, corrupted, contain viruses, arrive late or incomplete.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmama.org/pipermail/openmama-dev/attachments/20150325/6a1d3525/attachment.html>

781 - 800 of 2306