Market data subscriptions using qpid broker


Frank Quinn <fquinn.ni@...>
 

Hi Nestor,

Yes this is a strange one - you're right I could only recreate on a transient basis and only with the broker. I have put in a fix which seems to have prevented this behaviour in my environment by adding more forgiving timeouts particularly for the sending of subscription messages which I think may have been getting aborted in-flight.

Could you try the latest code off the OpenMAMA next branch and see if this helps in your environment?

Cheers,
Frank

On Tue, Apr 26, 2016 at 4:27 PM, Macrux <kmacrux@...> wrote:
Hi all,

I've been working with OpenMAMA 2.4.0 and the new Qpid Broker integration with the Java Qpid Broker implementation in Ubuntu 14.04. I've been experiencing some issues when I try to publish market data using capturereplayc. I use the next command

capturereplayc -S TEST -m qpid -tport broker -dictionary data/dictionaries/data.dict -f data/playbacks/openmama_utpcasheuro_capture.5000.10.qpid.mplay -r -v -v -v -v

Issues arise when I run mamalistenc or MamaListen.java example. For instance, with the next arguments they
can't get the dictionary:

mamalistenc -m qpid -tport broker -S TEST -s DE000CM95AU4.EUR.XPAR -v -v -v -v

I got this in mamalistenc console:

...
016-04-26 09:32:44: (15a53740) : deactivateAction(): (WOMBAT.DATA_DICT (DATA_DICT) (0x1152270)): mSubscBridge=(nil)
2016-04-26 09:32:44: (15a53740) : Subscription 0x1152270 is now at state MAMA_SUBSCRIPTION_DEACTIVATED.
2016-04-26 09:32:44: (15a53740) : imageRequest::timeoutCallback (): DATA_DICT: Timeout waiting for recap or initial value (subscription=0x1152270)
2016-04-26 09:32:44: (15a53740) : imageRequest::timeoutCallback (): source=WOMBAT; symbol=DATA_DICT
Timed out waiting for dictionary
Could not create dictionary.

And in the capturereplayc console, I got this:

...
2016-04-26 09:35:50: (f5dbf740) : Starting refresh message mini-cycle
2016-04-26 09:35:50: (f5dbf740) : No refreshes to send
2016-04-26 09:35:56: (f2552700) : qpidBridgeMamaTransportImpl_dispatchThread(): recv-ed
2016-04-26 09:35:56: (f5dbf740) : mamaSubscription_processMsg(): _MDDD.WOMBAT.DATA_DICT: msg = {{MdSubscSymbol,71,DATA_DICT},{MdSubscriptionType,60,4},{MdSubscMsgType,61,5},{MamaAppDataType,17,0},{MdSubscType,5,5},{MdSubscSourceHost,63,127.0.1.1}} subsc (0xacb940)


So I have to read the dictionary directly from file:

mamalistenc -m qpid -tport broker -S TEST -s DE000CM95AU4.EUR.XPAR -v -v -v -v -use_dict_file /home/nmarin/Apps/openmama-2.4.0/data/dictionaries/data.dict

Sometimes It just doesn't work and I got this message:

...
2016-04-26 09:46:53: (cd6a7740) : deactivateAction(): (TEST.DE000CM95AU4.EUR.XPAR (DE000CM95AU4.EUR.XPAR) (0x11113c0)): mSubscBridge=0x112b620
2016-04-26 09:46:53: (cd6a7740) : Subscription 0x11113c0 is now at state MAMA_SUBSCRIPTION_DEACTIVATED.
2016-04-26 09:46:53: (cd6a7740) : imageRequest::timeoutCallback (): DE000CM95AU4.EUR.XPAR: Timeout waiting for recap or initial value (subscription=0x11113c0)
2016-04-26 09:46:53: (cd6a7740) : imageRequest::timeoutCallback (): source=TEST; symbol=DE000CM95AU4.EUR.XPAR
An error occurred creating subscription for DE000CM95AU4.EUR.XPAR: STATUS_TIMEOUT


But sometimes if I restart capturereplayc and mamalistenc it works well. I'm actually a little confused with this behaviour, because sometimes it works, but sometimes doesn't work at all and I have to restart the broker, the capturereplayc or the mamalisten until this works. I've tested using regular transport (pub/sub) and it works perfect, so I think the issues show up only with the broker. I've also tested simple publish and subscribe with mamapublisherc, mamasubscriberc and it works fine.

Thanks for your help.

Nestor





_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev



macrux
 

Hi Frank, everything is running on Ubuntu 14.04, on the same machine. Qpid proton 0.9 and Qpid broker 6.0.1. Thanks.

Regards,

Nestor


On 26 April 2016 at 11:28, Frank Quinn <fquinn.ni@...> wrote:

Hi Nestor,

What versions of the Qpid broker and proton libraries are you using? Everything running on Ubuntu?

Cheers,
Frank


On Tue, 26 Apr 2016 16:27 Macrux, <kmacrux@...> wrote:
Hi all,

I've been working with OpenMAMA 2.4.0 and the new Qpid Broker integration with the Java Qpid Broker implementation in Ubuntu 14.04. I've been experiencing some issues when I try to publish market data using capturereplayc. I use the next command

capturereplayc -S TEST -m qpid -tport broker -dictionary data/dictionaries/data.dict -f data/playbacks/openmama_utpcasheuro_capture.5000.10.qpid.mplay -r -v -v -v -v

Issues arise when I run mamalistenc or MamaListen.java example. For instance, with the next arguments they
can't get the dictionary:

mamalistenc -m qpid -tport broker -S TEST -s DE000CM95AU4.EUR.XPAR -v -v -v -v

I got this in mamalistenc console:

...
016-04-26 09:32:44: (15a53740) : deactivateAction(): (WOMBAT.DATA_DICT (DATA_DICT) (0x1152270)): mSubscBridge=(nil)
2016-04-26 09:32:44: (15a53740) : Subscription 0x1152270 is now at state MAMA_SUBSCRIPTION_DEACTIVATED.
2016-04-26 09:32:44: (15a53740) : imageRequest::timeoutCallback (): DATA_DICT: Timeout waiting for recap or initial value (subscription=0x1152270)
2016-04-26 09:32:44: (15a53740) : imageRequest::timeoutCallback (): source=WOMBAT; symbol=DATA_DICT
Timed out waiting for dictionary
Could not create dictionary.

And in the capturereplayc console, I got this:

...
2016-04-26 09:35:50: (f5dbf740) : Starting refresh message mini-cycle
2016-04-26 09:35:50: (f5dbf740) : No refreshes to send
2016-04-26 09:35:56: (f2552700) : qpidBridgeMamaTransportImpl_dispatchThread(): recv-ed
2016-04-26 09:35:56: (f5dbf740) : mamaSubscription_processMsg(): _MDDD.WOMBAT.DATA_DICT: msg = {{MdSubscSymbol,71,DATA_DICT},{MdSubscriptionType,60,4},{MdSubscMsgType,61,5},{MamaAppDataType,17,0},{MdSubscType,5,5},{MdSubscSourceHost,63,127.0.1.1}} subsc (0xacb940)


So I have to read the dictionary directly from file:

mamalistenc -m qpid -tport broker -S TEST -s DE000CM95AU4.EUR.XPAR -v -v -v -v -use_dict_file /home/nmarin/Apps/openmama-2.4.0/data/dictionaries/data.dict

Sometimes It just doesn't work and I got this message:

...
2016-04-26 09:46:53: (cd6a7740) : deactivateAction(): (TEST.DE000CM95AU4.EUR.XPAR (DE000CM95AU4.EUR.XPAR) (0x11113c0)): mSubscBridge=0x112b620
2016-04-26 09:46:53: (cd6a7740) : Subscription 0x11113c0 is now at state MAMA_SUBSCRIPTION_DEACTIVATED.
2016-04-26 09:46:53: (cd6a7740) : imageRequest::timeoutCallback (): DE000CM95AU4.EUR.XPAR: Timeout waiting for recap or initial value (subscription=0x11113c0)
2016-04-26 09:46:53: (cd6a7740) : imageRequest::timeoutCallback (): source=TEST; symbol=DE000CM95AU4.EUR.XPAR
An error occurred creating subscription for DE000CM95AU4.EUR.XPAR: STATUS_TIMEOUT


But sometimes if I restart capturereplayc and mamalistenc it works well. I'm actually a little confused with this behaviour, because sometimes it works, but sometimes doesn't work at all and I have to restart the broker, the capturereplayc or the mamalisten until this works. I've tested using regular transport (pub/sub) and it works perfect, so I think the issues show up only with the broker. I've also tested simple publish and subscribe with mamapublisherc, mamasubscriberc and it works fine.

Thanks for your help.

Nestor




_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev


Frank Quinn <fquinn.ni@...>
 

Hi Nestor,

What versions of the Qpid broker and proton libraries are you using? Everything running on Ubuntu?

Cheers,
Frank


On Tue, 26 Apr 2016 16:27 Macrux, <kmacrux@...> wrote:
Hi all,

I've been working with OpenMAMA 2.4.0 and the new Qpid Broker integration with the Java Qpid Broker implementation in Ubuntu 14.04. I've been experiencing some issues when I try to publish market data using capturereplayc. I use the next command

capturereplayc -S TEST -m qpid -tport broker -dictionary data/dictionaries/data.dict -f data/playbacks/openmama_utpcasheuro_capture.5000.10.qpid.mplay -r -v -v -v -v

Issues arise when I run mamalistenc or MamaListen.java example. For instance, with the next arguments they
can't get the dictionary:

mamalistenc -m qpid -tport broker -S TEST -s DE000CM95AU4.EUR.XPAR -v -v -v -v

I got this in mamalistenc console:

...
016-04-26 09:32:44: (15a53740) : deactivateAction(): (WOMBAT.DATA_DICT (DATA_DICT) (0x1152270)): mSubscBridge=(nil)
2016-04-26 09:32:44: (15a53740) : Subscription 0x1152270 is now at state MAMA_SUBSCRIPTION_DEACTIVATED.
2016-04-26 09:32:44: (15a53740) : imageRequest::timeoutCallback (): DATA_DICT: Timeout waiting for recap or initial value (subscription=0x1152270)
2016-04-26 09:32:44: (15a53740) : imageRequest::timeoutCallback (): source=WOMBAT; symbol=DATA_DICT
Timed out waiting for dictionary
Could not create dictionary.

And in the capturereplayc console, I got this:

...
2016-04-26 09:35:50: (f5dbf740) : Starting refresh message mini-cycle
2016-04-26 09:35:50: (f5dbf740) : No refreshes to send
2016-04-26 09:35:56: (f2552700) : qpidBridgeMamaTransportImpl_dispatchThread(): recv-ed
2016-04-26 09:35:56: (f5dbf740) : mamaSubscription_processMsg(): _MDDD.WOMBAT.DATA_DICT: msg = {{MdSubscSymbol,71,DATA_DICT},{MdSubscriptionType,60,4},{MdSubscMsgType,61,5},{MamaAppDataType,17,0},{MdSubscType,5,5},{MdSubscSourceHost,63,127.0.1.1}} subsc (0xacb940)


So I have to read the dictionary directly from file:

mamalistenc -m qpid -tport broker -S TEST -s DE000CM95AU4.EUR.XPAR -v -v -v -v -use_dict_file /home/nmarin/Apps/openmama-2.4.0/data/dictionaries/data.dict

Sometimes It just doesn't work and I got this message:

...
2016-04-26 09:46:53: (cd6a7740) : deactivateAction(): (TEST.DE000CM95AU4.EUR.XPAR (DE000CM95AU4.EUR.XPAR) (0x11113c0)): mSubscBridge=0x112b620
2016-04-26 09:46:53: (cd6a7740) : Subscription 0x11113c0 is now at state MAMA_SUBSCRIPTION_DEACTIVATED.
2016-04-26 09:46:53: (cd6a7740) : imageRequest::timeoutCallback (): DE000CM95AU4.EUR.XPAR: Timeout waiting for recap or initial value (subscription=0x11113c0)
2016-04-26 09:46:53: (cd6a7740) : imageRequest::timeoutCallback (): source=TEST; symbol=DE000CM95AU4.EUR.XPAR
An error occurred creating subscription for DE000CM95AU4.EUR.XPAR: STATUS_TIMEOUT


But sometimes if I restart capturereplayc and mamalistenc it works well. I'm actually a little confused with this behaviour, because sometimes it works, but sometimes doesn't work at all and I have to restart the broker, the capturereplayc or the mamalisten until this works. I've tested using regular transport (pub/sub) and it works perfect, so I think the issues show up only with the broker. I've also tested simple publish and subscribe with mamapublisherc, mamasubscriberc and it works fine.

Thanks for your help.

Nestor




_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev


macrux
 

Hi all,

I've been working with OpenMAMA 2.4.0 and the new Qpid Broker integration with the Java Qpid Broker implementation in Ubuntu 14.04. I've been experiencing some issues when I try to publish market data using capturereplayc. I use the next command

capturereplayc -S TEST -m qpid -tport broker -dictionary data/dictionaries/data.dict -f data/playbacks/openmama_utpcasheuro_capture.5000.10.qpid.mplay -r -v -v -v -v

Issues arise when I run mamalistenc or MamaListen.java example. For instance, with the next arguments they
can't get the dictionary:

mamalistenc -m qpid -tport broker -S TEST -s DE000CM95AU4.EUR.XPAR -v -v -v -v

I got this in mamalistenc console:

...
016-04-26 09:32:44: (15a53740) : deactivateAction(): (WOMBAT.DATA_DICT (DATA_DICT) (0x1152270)): mSubscBridge=(nil)
2016-04-26 09:32:44: (15a53740) : Subscription 0x1152270 is now at state MAMA_SUBSCRIPTION_DEACTIVATED.
2016-04-26 09:32:44: (15a53740) : imageRequest::timeoutCallback (): DATA_DICT: Timeout waiting for recap or initial value (subscription=0x1152270)
2016-04-26 09:32:44: (15a53740) : imageRequest::timeoutCallback (): source=WOMBAT; symbol=DATA_DICT
Timed out waiting for dictionary
Could not create dictionary.

And in the capturereplayc console, I got this:

...
2016-04-26 09:35:50: (f5dbf740) : Starting refresh message mini-cycle
2016-04-26 09:35:50: (f5dbf740) : No refreshes to send
2016-04-26 09:35:56: (f2552700) : qpidBridgeMamaTransportImpl_dispatchThread(): recv-ed
2016-04-26 09:35:56: (f5dbf740) : mamaSubscription_processMsg(): _MDDD.WOMBAT.DATA_DICT: msg = {{MdSubscSymbol,71,DATA_DICT},{MdSubscriptionType,60,4},{MdSubscMsgType,61,5},{MamaAppDataType,17,0},{MdSubscType,5,5},{MdSubscSourceHost,63,127.0.1.1}} subsc (0xacb940)


So I have to read the dictionary directly from file:

mamalistenc -m qpid -tport broker -S TEST -s DE000CM95AU4.EUR.XPAR -v -v -v -v -use_dict_file /home/nmarin/Apps/openmama-2.4.0/data/dictionaries/data.dict

Sometimes It just doesn't work and I got this message:

...
2016-04-26 09:46:53: (cd6a7740) : deactivateAction(): (TEST.DE000CM95AU4.EUR.XPAR (DE000CM95AU4.EUR.XPAR) (0x11113c0)): mSubscBridge=0x112b620
2016-04-26 09:46:53: (cd6a7740) : Subscription 0x11113c0 is now at state MAMA_SUBSCRIPTION_DEACTIVATED.
2016-04-26 09:46:53: (cd6a7740) : imageRequest::timeoutCallback (): DE000CM95AU4.EUR.XPAR: Timeout waiting for recap or initial value (subscription=0x11113c0)
2016-04-26 09:46:53: (cd6a7740) : imageRequest::timeoutCallback (): source=TEST; symbol=DE000CM95AU4.EUR.XPAR
An error occurred creating subscription for DE000CM95AU4.EUR.XPAR: STATUS_TIMEOUT


But sometimes if I restart capturereplayc and mamalistenc it works well. I'm actually a little confused with this behaviour, because sometimes it works, but sometimes doesn't work at all and I have to restart the broker, the capturereplayc or the mamalisten until this works. I've tested using regular transport (pub/sub) and it works perfect, so I think the issues show up only with the broker. I've also tested simple publish and subscribe with mamapublisherc, mamasubscriberc and it works fine.

Thanks for your help.

Nestor