Openmama-dev@lists.openmama.org

Home Messages Hashtags Subgroups

Expanded

Between

and

Error handling in market data subscription activation and error notification to application Alireza Assadzadeh <Alireza.Assadzadeh@...> #1399 Hi Folks, I have noticed that in Mama C subscription, if a market data subscription activation callback (i.e. createAction) fails the error is not propagated in Mama subscription layer to the application through the onError user callback for the subscription. For example, a throttled market data subscription may fail to activate due to Middleware Bridge create subscription failure. Specifically for the Middleware Bridge failure case, looking at mamaSubscriptionImpl_completeMarketDataInitialisation, the call to mamaSubscription_initialize may have failed for example in birdgeMamaSubscriptionCreate call. The return code is not examined in the Mama Subscription and the subscription is always transitioned to activated state and the onCreate user callback is called. The onCreate user callback is called because the subscription creation (activation) is completed from the creation throttle. This seems to be the design intent and there is no expectation that the Mama subscription would be calling the onError user callback in such failure cases. Can you please confirm and/or let me know your opinion on this? Assuming this is the design intent, I think in the Middleware Bridge create subscription failure case, it follows that it is the responsibility of Middleware Bridge to notify the Mama C layer and the application about such errors asynchronously (beyond the function return status code for birdgeMamaSubscriptionCreate). The Middleware Bridge can provide the error notification through a subscription queue event error callback which will then perform the Mama subscription deactivation (optionally) and the call to the user onError callback. In other words, the Middleware Bridge may enqueue an event for mamaSubscription_processErr or a similar callback of its own to deactivate the Mama subscription (optionally) and call the user onError callback. Note that there are cases where the Middleware Bridge may determine immediately in the bridge subscription create call that it is going to fail the subscription create call. There are also other cases where the Middleware Bridge (depending on its capabilities) may determine at later point in time, in an asynchronously fashion, that the Middleware Bridge subscription creation process encountered an error subsequently. So, for such cases, the Middleware Bridge may already have a mechanism in place to deactivate the Mama subscription and call the user onError callback. Regards, --Alireza
More All Messages By This Member
Re: Asynchronous publisher events Gary Molloy <g.molloy@...> #1398 Hi Guys, Thanks for the feedback on the proposed dates. From the feedback I think the best date that suits most would be Tuesday 16th December @ 4pm UTC, so let's schedule the call for then. The connection details once again are: Conference call details: UK Number: +44-28-90998719 US Number: +646-828-7349 Conference ID: 4741664 Looking forward to the call. Thanks, Gary Gary Molloy – SR Labs Adelaide Exchange \| 24-26 Adelaide Street \| Belfast \| BT2 8GD Tel: +44 28 9099 7580 Ext 3397 g.molloy@... toggle quoted message Show quoted text -----Original Message----- From: Gary Molloy Sent: 08 December 2014 11:16 To: Glenn McClements; Sam Wilson; Tom Doust; openmama-dev@... Subject: RE: [Openmama-dev] Asynchronous publisher events Hi Guys, I would like to propose a call this week (or next week) to kick off the discussion for the new mamaPublisher event handling feature. I was hoping to schedule a call either Thursday 11th, or Tuesday 16th of December @ 4pm UTC. I would appreciate it if you could let us know which date would suit most? Conference call details: UK Number: +44-28-90998719 US Number: +646-828-7349 Conference ID: 4741664 Thanks Gary Gary Molloy – SR Labs Adelaide Exchange \| 24-26 Adelaide Street \| Belfast \| BT2 8GD Tel: +44 28 9099 7580 Ext 3397 g.molloy@... -----Original Message----- From: Gary Molloy Sent: 02 December 2014 17:44 To: Glenn McClements; Sam Wilson; Tom Doust; openmama-dev@... Subject: RE: [Openmama-dev] Asynchronous publisher events Hi Guys, I have created an appraisal / proposal document for the new publisher event handling feature on Google Docs which you can find here: https://docs.google.com/document/d/1bxy0Muxit4uc8Bz-y6CS3wFq5I7sJ0qAkA8-zr5vLec/edit?usp=sharing If you would like to review the document and we can then schedule a call so we can go over the proposal to discuss it in further detail. Thanks, Gary Gary Molloy SR.LABS Proven High Speed Electronic Trading Solutions g.molloy@... -----Original Message----- From: Glenn McClements Sent: 13 November 2014 12:15 To: Sam Wilson; Tom Doust; openmama-dev@... Cc: Gary Molloy Subject: Re: [Openmama-dev] Asynchronous publisher events Yeah I think something analogous to mamaMsgCallbacks (which should have been called mamaSubscriberCallbacks in hindsight) for the mamaPublisher is the nicest fit here. The advantage of doing it this was is that the set of callbacks can be optional and extensible. One gotcha though is the question of which thread/queue the callback should be fired on? Currently the mamaPublisher doesn’t have a queue associated with it so we’ll probably need to add this as well. For next steps I propose we collate the requirements for publisher events and put together a proposal/appraisal, then we can figure out who wants to implement and test it. Possible events are: - undeliverable (with a reason/status) - delivered - …. anything else? Gary on our side will pull these together and we can set up a call to discuss. Cheers, Glenn On 11/11/2014 16:45, "Sam Wilson" <Sam.Wilson@...> wrote: Hey Glenn, Tom, I have to agree with Glenn here. I don't think these kind of events belong alongside regular messages, for a couple of reasons. For one, simply enough, they aren't messages. They don't originate from a publisher, they wouldn't travel along the wire, and they probably would have to be synthesized in the middleware bridge. Secondly, Tom's solution only works for sendFromInbox. The callback mechanism is much more general, and would work for non-inbox messages as well, which is important for our use case. Assuming we do go ahead with implementing callbacks for asynchronous publisher events, what needs to be thought about/discussed before moving on to writing/accepting a patch? I have two ideas on how to implement this, but keep in mind that I'm rather new to OpenMAMA and market data in general. We could use a mamaTransportTopic event, and report the errors with a callback on the transport. The other option that I see, which Glenn hinted at, is that we create a structure similar to mamaMsgCallbacks, called mamaPublisherCallbacks; and a pair of new methods mamaPublisher_createWithCB and/or mamaPublisher_setCallbacks. That way we report the error on the specific publisher. What are your thoughts? Regards, Sam On 14-11-07 10:57 AM, Glenn McClements wrote: Hi Tom, Sam, Before you responded I was considering the use of an inbox as a solution to Sam’s query but I wasn’t overly keen on it. The reason being that being rejected by the broker felt very much like a middleware level event, not an application or market data event so it feels better to me that the error comes from the middleware as a callback. Also, responding with a MamaMsg to the sender means assuming the connection is up and alive, whereas a middleware ACL policy might actually prevent this from happening. It could be that you form the message on the client side in the bridge, but this itself feels incorrect as it’s translating from a middleware level even into a data message. (Also a minor point is that the MAMA_MSG_TYPE_SEC_STATUS relates to security status events on a exchange). This is not to say that a message response is incorrect for RMDS posts however, it’s just a different type of event. For Sam’s point I do prefer the idea of currently returning a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and the transport is currently the best/only place to do this. Going forward we could add callbacks to the MamaPublisher object to inform the client asynchronously of events like this, and MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED seems reasonable though we would need to consider how this would be used/interact with the higher level concept of market data entitlements. Another related concept is guaranteed or acknowledged messaging, which would generate other asynchronous event on a publishers. Like the original issue, this could be done at a high level with MamaMsgs being passed back to acknowledge delivery, but generally this is implemented at the middleware level so again a publisher callback would be better (with a handle to the original message as well for context). Glenn On 07/11/2014 14:20, "Tom Doust" <tom.doust@...> wrote: Hi In the Tick42 RMDS bridge we address this problem by allowing applications to use the sendFromInbox methods on a publisher and where applicable converting the an asynchronous response from the RMDS into a mama message which is delivered to the inbox. Although this is intended to support the RMDS message "post" publishing model that will ACK (or NAK) every message I think it will work for any middleware that has an asynchronous response mechanism; other publishing models on RMDS do not have a message reponse. The message sent to the inbox is arbitrary but it would make sense if other bridges that used this technique used the same message structure. The message we use is formed as follows mamaMsg_addI32(msg, "MdMsgType", 1, MAMA_MSG_TYPE_SEC_STATUS); mamaMsgStatus status = MAMA_MSG_STATUS_OK; if(nakCode != RSSL_NAKC_NONE) { status = RsslNakCode2MamaMsgStatus(nakCode); } mamaMsg_addI32(msg, "MdMsgStatus", 2, status); mamaMsg_addString(msg, "wSymbol",470, symbol_.c_str()); As you can see we just convert an rssl NAK code to an equivalent mama code, or use an OK status in the case of an ACK. These include states like MAMA_MSG_STATUS_NOT_ENTITLED and MAMA_MSG_STATUS_BAD_SYMBOL amongst a number of others. Obviously the set of code and the mappings are platform specific The one thing I think is missing here is a field containing a text string which could carry more information relevant to the client application. We would propose that everyone addressing the issue of processing asynchronous publishing responses takes this approach and that as a community we agree on a convention for the message content. Best Regards Tom Doust -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Sam Wilson Sent: 06 November 2014 7:45 PM To: openmama-dev@... Subject: [Openmama-dev] Asynchronous publisher events Hey all, We're looking for a way to notify a mama application asynchronously of errors while publishing, specifically when the published message was rejected by the message broker's ACL. Our API assumes the messages were accepted, and returns immediately. After some time you might get a connection-level event callback informing the application that its messages were rejected. I have two questions. First off, what is the best way to represent this situation in 2.3.1? We've been tossing around the idea of just tearing down the connection, giving the application a MAMA_TRANSPORT_DISCONNECT or a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and logging the error. Since the application or the environment is incorrectly configured, there's not much that can be done by the application anyways. Secondly, if openmama were to be extended in the future to support these kind of events, what would such extensions look like? Perhaps adding a MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED to the mamaTransportTopicEvent enumeration? Thanks, Sam _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev
More All Messages By This Member
Re: Asynchronous publisher events Alpert, Reed <reed.alpert@...> #1397 16th is better for me. Thanks, Reed. Reed Alpert \| Corporate & Investment Bank \| Market Data Services \| J.P. Morgan \| 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 \| T: 718.242.5198 \| M: 917.414.4613 \| reed.alpert@... Alternate Contact: CIB PIM Trading Technology Solutions NA \| CIB_PIM_Trading_Technology_Solutions_NA@... toggle quoted message Show quoted text -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Gary Molloy Sent: Monday, December 08, 2014 6:16 AM To: Glenn McClements; Sam Wilson; Tom Doust; openmama-dev@... Subject: Re: [Openmama-dev] Asynchronous publisher events Hi Guys, I would like to propose a call this week (or next week) to kick off the discussion for the new mamaPublisher event handling feature. I was hoping to schedule a call either Thursday 11th, or Tuesday 16th of December @ 4pm UTC. I would appreciate it if you could let us know which date would suit most? Conference call details: UK Number: +44-28-90998719 US Number: +646-828-7349 Conference ID: 4741664 Thanks Gary Gary Molloy – SR Labs Adelaide Exchange \| 24-26 Adelaide Street \| Belfast \| BT2 8GD Tel: +44 28 9099 7580 Ext 3397 g.molloy@... -----Original Message----- From: Gary Molloy Sent: 02 December 2014 17:44 To: Glenn McClements; Sam Wilson; Tom Doust; openmama-dev@... Subject: RE: [Openmama-dev] Asynchronous publisher events Hi Guys, I have created an appraisal / proposal document for the new publisher event handling feature on Google Docs which you can find here: https://docs.google.com/document/d/1bxy0Muxit4uc8Bz-y6CS3wFq5I7sJ0qAkA8-zr5vLec/edit?usp=sharing If you would like to review the document and we can then schedule a call so we can go over the proposal to discuss it in further detail. Thanks, Gary Gary Molloy SR.LABS Proven High Speed Electronic Trading Solutions g.molloy@... -----Original Message----- From: Glenn McClements Sent: 13 November 2014 12:15 To: Sam Wilson; Tom Doust; openmama-dev@... Cc: Gary Molloy Subject: Re: [Openmama-dev] Asynchronous publisher events Yeah I think something analogous to mamaMsgCallbacks (which should have been called mamaSubscriberCallbacks in hindsight) for the mamaPublisher is the nicest fit here. The advantage of doing it this was is that the set of callbacks can be optional and extensible. One gotcha though is the question of which thread/queue the callback should be fired on? Currently the mamaPublisher doesn’t have a queue associated with it so we’ll probably need to add this as well. For next steps I propose we collate the requirements for publisher events and put together a proposal/appraisal, then we can figure out who wants to implement and test it. Possible events are: - undeliverable (with a reason/status) - delivered - …. anything else? Gary on our side will pull these together and we can set up a call to discuss. Cheers, Glenn On 11/11/2014 16:45, "Sam Wilson" <Sam.Wilson@...> wrote: Hey Glenn, Tom, I have to agree with Glenn here. I don't think these kind of events belong alongside regular messages, for a couple of reasons. For one, simply enough, they aren't messages. They don't originate from a publisher, they wouldn't travel along the wire, and they probably would have to be synthesized in the middleware bridge. Secondly, Tom's solution only works for sendFromInbox. The callback mechanism is much more general, and would work for non-inbox messages as well, which is important for our use case. Assuming we do go ahead with implementing callbacks for asynchronous publisher events, what needs to be thought about/discussed before moving on to writing/accepting a patch? I have two ideas on how to implement this, but keep in mind that I'm rather new to OpenMAMA and market data in general. We could use a mamaTransportTopic event, and report the errors with a callback on the transport. The other option that I see, which Glenn hinted at, is that we create a structure similar to mamaMsgCallbacks, called mamaPublisherCallbacks; and a pair of new methods mamaPublisher_createWithCB and/or mamaPublisher_setCallbacks. That way we report the error on the specific publisher. What are your thoughts? Regards, Sam On 14-11-07 10:57 AM, Glenn McClements wrote: Hi Tom, Sam, Before you responded I was considering the use of an inbox as a solution to Sam’s query but I wasn’t overly keen on it. The reason being that being rejected by the broker felt very much like a middleware level event, not an application or market data event so it feels better to me that the error comes from the middleware as a callback. Also, responding with a MamaMsg to the sender means assuming the connection is up and alive, whereas a middleware ACL policy might actually prevent this from happening. It could be that you form the message on the client side in the bridge, but this itself feels incorrect as it’s translating from a middleware level even into a data message. (Also a minor point is that the MAMA_MSG_TYPE_SEC_STATUS relates to security status events on a exchange). This is not to say that a message response is incorrect for RMDS posts however, it’s just a different type of event. For Sam’s point I do prefer the idea of currently returning a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and the transport is currently the best/only place to do this. Going forward we could add callbacks to the MamaPublisher object to inform the client asynchronously of events like this, and MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED seems reasonable though we would need to consider how this would be used/interact with the higher level concept of market data entitlements. Another related concept is guaranteed or acknowledged messaging, which would generate other asynchronous event on a publishers. Like the original issue, this could be done at a high level with MamaMsgs being passed back to acknowledge delivery, but generally this is implemented at the middleware level so again a publisher callback would be better (with a handle to the original message as well for context). Glenn On 07/11/2014 14:20, "Tom Doust" <tom.doust@...> wrote: Hi In the Tick42 RMDS bridge we address this problem by allowing applications to use the sendFromInbox methods on a publisher and where applicable converting the an asynchronous response from the RMDS into a mama message which is delivered to the inbox. Although this is intended to support the RMDS message "post" publishing model that will ACK (or NAK) every message I think it will work for any middleware that has an asynchronous response mechanism; other publishing models on RMDS do not have a message reponse. The message sent to the inbox is arbitrary but it would make sense if other bridges that used this technique used the same message structure. The message we use is formed as follows mamaMsg_addI32(msg, "MdMsgType", 1, MAMA_MSG_TYPE_SEC_STATUS); mamaMsgStatus status = MAMA_MSG_STATUS_OK; if(nakCode != RSSL_NAKC_NONE) { status = RsslNakCode2MamaMsgStatus(nakCode); } mamaMsg_addI32(msg, "MdMsgStatus", 2, status); mamaMsg_addString(msg, "wSymbol",470, symbol_.c_str()); As you can see we just convert an rssl NAK code to an equivalent mama code, or use an OK status in the case of an ACK. These include states like MAMA_MSG_STATUS_NOT_ENTITLED and MAMA_MSG_STATUS_BAD_SYMBOL amongst a number of others. Obviously the set of code and the mappings are platform specific The one thing I think is missing here is a field containing a text string which could carry more information relevant to the client application. We would propose that everyone addressing the issue of processing asynchronous publishing responses takes this approach and that as a community we agree on a convention for the message content. Best Regards Tom Doust -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Sam Wilson Sent: 06 November 2014 7:45 PM To: openmama-dev@... Subject: [Openmama-dev] Asynchronous publisher events Hey all, We're looking for a way to notify a mama application asynchronously of errors while publishing, specifically when the published message was rejected by the message broker's ACL. Our API assumes the messages were accepted, and returns immediately. After some time you might get a connection-level event callback informing the application that its messages were rejected. I have two questions. First off, what is the best way to represent this situation in 2.3.1? We've been tossing around the idea of just tearing down the connection, giving the application a MAMA_TRANSPORT_DISCONNECT or a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and logging the error. Since the application or the environment is incorrectly configured, there's not much that can be done by the application anyways. Secondly, if openmama were to be extended in the future to support these kind of events, what would such extensions look like? Perhaps adding a MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED to the mamaTransportTopicEvent enumeration? Thanks, Sam _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.
More All Messages By This Member
Re: Asynchronous publisher events Alireza Assadzadeh <Alireza.Assadzadeh@...> #1396 Either one works for me. Looking forward to the meeting. --Alireza toggle quoted message Show quoted text -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama- dev-bounces@...] On Behalf Of Gary Molloy Sent: Monday, December 08, 2014 6:16 AM To: Glenn McClements; Sam Wilson; Tom Doust; openmama- dev@... Subject: Re: [Openmama-dev] Asynchronous publisher events Hi Guys, I would like to propose a call this week (or next week) to kick off the discussion for the new mamaPublisher event handling feature. I was hoping to schedule a call either Thursday 11th, or Tuesday 16th of December @ 4pm UTC. I would appreciate it if you could let us know which date would suit most? Conference call details: UK Number: +44-28-90998719 US Number: +646-828-7349 Conference ID: 4741664 Thanks Gary Gary Molloy – SR Labs Adelaide Exchange \| 24-26 Adelaide Street \| Belfast \| BT2 8GD Tel: +44 28 9099 7580 Ext 3397 g.molloy@... -----Original Message----- From: Gary Molloy Sent: 02 December 2014 17:44 To: Glenn McClements; Sam Wilson; Tom Doust; openmama- dev@... Subject: RE: [Openmama-dev] Asynchronous publisher events Hi Guys, I have created an appraisal / proposal document for the new publisher event handling feature on Google Docs which you can find here: https://docs.google.com/document/d/1bxy0Muxit4uc8Bz- y6CS3wFq5I7sJ0qAkA8-zr5vLec/edit?usp=sharing If you would like to review the document and we can then schedule a call so we can go over the proposal to discuss it in further detail. Thanks, Gary Gary Molloy SR.LABS Proven High Speed Electronic Trading Solutions g.molloy@... -----Original Message----- From: Glenn McClements Sent: 13 November 2014 12:15 To: Sam Wilson; Tom Doust; openmama-dev@... Cc: Gary Molloy Subject: Re: [Openmama-dev] Asynchronous publisher events Yeah I think something analogous to mamaMsgCallbacks (which should have been called mamaSubscriberCallbacks in hindsight) for the mamaPublisher is the nicest fit here. The advantage of doing it this was is that the set of callbacks can be optional and extensible. One gotcha though is the question of which thread/queue the callback should be fired on? Currently the mamaPublisher doesn’t have a queue associated with it so we’ll probably need to add this as well. For next steps I propose we collate the requirements for publisher events and put together a proposal/appraisal, then we can figure out who wants to implement and test it. Possible events are: - undeliverable (with a reason/status) - delivered - …. anything else? Gary on our side will pull these together and we can set up a call to discuss. Cheers, Glenn On 11/11/2014 16:45, "Sam Wilson" <Sam.Wilson@...> wrote: Hey Glenn, Tom, I have to agree with Glenn here. I don't think these kind of events belong alongside regular messages, for a couple of reasons. For one, simply enough, they aren't messages. They don't originate from a publisher, they wouldn't travel along the wire, and they probably would have to be synthesized in the middleware bridge. Secondly, Tom's solution only works for sendFromInbox. The callback mechanism is much more general, and would work for non-inbox messages as well, which is important for our use case. Assuming we do go ahead with implementing callbacks for asynchronous publisher events, what needs to be thought about/discussed before moving on to writing/accepting a patch? I have two ideas on how to implement this, but keep in mind that I'm rather new to OpenMAMA and market data in general. We could use a mamaTransportTopic event, and report the errors with a callback on the transport. The other option that I see, which Glenn hinted at, is that we create a structure similar to mamaMsgCallbacks, called mamaPublisherCallbacks; and a pair of new methods mamaPublisher_createWithCB and/or mamaPublisher_setCallbacks. That way we report the error on the specific publisher. What are your thoughts? Regards, Sam On 14-11-07 10:57 AM, Glenn McClements wrote: Hi Tom, Sam, Before you responded I was considering the use of an inbox as a solution to Sam’s query but I wasn’t overly keen on it. The reason being that being rejected by the broker felt very much like a middleware level event, not an application or market data event so it feels better to me that the error comes from the middleware as a callback. Also, responding with a MamaMsg to the sender means assuming the connection is up and alive, whereas a middleware ACL policy might actually prevent this from happening. It could be that you form the message on the client side in the bridge, but this itself feels incorrect as it’s translating from a middleware level even into a data message. (Also a minor point is that the MAMA_MSG_TYPE_SEC_STATUS relates to security status events on a exchange). This is not to say that a message response is incorrect for RMDS posts however, it’s just a different type of event. For Sam’s point I do prefer the idea of currently returning a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and the transport is currently the best/only place to do this. Going forward we could add callbacks to the MamaPublisher object to inform the client asynchronously of events like this, and MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED seems reasonable though we would need to consider how this would be used/interact with the higher level concept of market data entitlements. Another related concept is guaranteed or acknowledged messaging, which would generate other asynchronous event on a publishers. Like the original issue, this could be done at a high level with MamaMsgs being passed back to acknowledge delivery, but generally this is implemented at the middleware level so again a publisher callback would be better (with a handle to the original message as well for context). Glenn On 07/11/2014 14:20, "Tom Doust" <tom.doust@...> wrote: Hi In the Tick42 RMDS bridge we address this problem by allowing applications to use the sendFromInbox methods on a publisher and where applicable converting the an asynchronous response from the RMDS into a mama message which is delivered to the inbox. Although this is intended to support the RMDS message "post" publishing model that will ACK (or NAK) every message I think it will work for any middleware that has an asynchronous response mechanism; other publishing models on RMDS do not have a message reponse. The message sent to the inbox is arbitrary but it would make sense if other bridges that used this technique used the same message structure. The message we use is formed as follows mamaMsg_addI32(msg, "MdMsgType", 1, MAMA_MSG_TYPE_SEC_STATUS); mamaMsgStatus status = MAMA_MSG_STATUS_OK; if(nakCode != RSSL_NAKC_NONE) { status = RsslNakCode2MamaMsgStatus(nakCode); } mamaMsg_addI32(msg, "MdMsgStatus", 2, status); mamaMsg_addString(msg, "wSymbol",470, symbol_.c_str()); As you can see we just convert an rssl NAK code to an equivalent mama code, or use an OK status in the case of an ACK. These include states like MAMA_MSG_STATUS_NOT_ENTITLED and MAMA_MSG_STATUS_BAD_SYMBOL amongst a number of others. Obviously the set of code and the mappings are platform specific The one thing I think is missing here is a field containing a text string which could carry more information relevant to the client application. We would propose that everyone addressing the issue of processing asynchronous publishing responses takes this approach and that as a community we agree on a convention for the message content. Best Regards Tom Doust -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Sam Wilson Sent: 06 November 2014 7:45 PM To: openmama-dev@... Subject: [Openmama-dev] Asynchronous publisher events Hey all, We're looking for a way to notify a mama application asynchronously of errors while publishing, specifically when the published message was rejected by the message broker's ACL. Our API assumes the messages were accepted, and returns immediately. After some time you might get a connection-level event callback informing the application that its messages were rejected. I have two questions. First off, what is the best way to represent this situation in 2.3.1? We've been tossing around the idea of just tearing down the connection, giving the application a MAMA_TRANSPORT_DISCONNECT or a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and logging the error. Since the application or the environment is incorrectly configured, there's not much that can be done by the application anyways. Secondly, if openmama were to be extended in the future to support these kind of events, what would such extensions look like? Perhaps adding a MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED to the mamaTransportTopicEvent enumeration? Thanks, Sam _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev
More All Messages By This Member
Re: Asynchronous publisher events Tom Doust #1394 Hi. Either is OK by me at the moment Tom toggle quoted message Show quoted text -----Original Message----- From: Gary Molloy [mailto:g.molloy@...] Sent: 08 December 2014 11:16 AM To: Glenn McClements; Sam Wilson; Tom Doust; openmama-dev@... Subject: RE: [Openmama-dev] Asynchronous publisher events Hi Guys, I would like to propose a call this week (or next week) to kick off the discussion for the new mamaPublisher event handling feature. I was hoping to schedule a call either Thursday 11th, or Tuesday 16th of December @ 4pm UTC. I would appreciate it if you could let us know which date would suit most? Conference call details: UK Number: +44-28-90998719 US Number: +646-828-7349 Conference ID: 4741664 Thanks Gary Gary Molloy – SR Labs Adelaide Exchange \| 24-26 Adelaide Street \| Belfast \| BT2 8GD Tel: +44 28 9099 7580 Ext 3397 g.molloy@... -----Original Message----- From: Gary Molloy Sent: 02 December 2014 17:44 To: Glenn McClements; Sam Wilson; Tom Doust; openmama-dev@... Subject: RE: [Openmama-dev] Asynchronous publisher events Hi Guys, I have created an appraisal / proposal document for the new publisher event handling feature on Google Docs which you can find here: https://docs.google.com/document/d/1bxy0Muxit4uc8Bz-y6CS3wFq5I7sJ0qAkA8-zr5vLec/edit?usp=sharing If you would like to review the document and we can then schedule a call so we can go over the proposal to discuss it in further detail. Thanks, Gary Gary Molloy SR.LABS Proven High Speed Electronic Trading Solutions g.molloy@... -----Original Message----- From: Glenn McClements Sent: 13 November 2014 12:15 To: Sam Wilson; Tom Doust; openmama-dev@... Cc: Gary Molloy Subject: Re: [Openmama-dev] Asynchronous publisher events Yeah I think something analogous to mamaMsgCallbacks (which should have been called mamaSubscriberCallbacks in hindsight) for the mamaPublisher is the nicest fit here. The advantage of doing it this was is that the set of callbacks can be optional and extensible. One gotcha though is the question of which thread/queue the callback should be fired on? Currently the mamaPublisher doesn’t have a queue associated with it so we’ll probably need to add this as well. For next steps I propose we collate the requirements for publisher events and put together a proposal/appraisal, then we can figure out who wants to implement and test it. Possible events are: - undeliverable (with a reason/status) - delivered - …. anything else? Gary on our side will pull these together and we can set up a call to discuss. Cheers, Glenn On 11/11/2014 16:45, "Sam Wilson" <Sam.Wilson@...> wrote: Hey Glenn, Tom, I have to agree with Glenn here. I don't think these kind of events belong alongside regular messages, for a couple of reasons. For one, simply enough, they aren't messages. They don't originate from a publisher, they wouldn't travel along the wire, and they probably would have to be synthesized in the middleware bridge. Secondly, Tom's solution only works for sendFromInbox. The callback mechanism is much more general, and would work for non-inbox messages as well, which is important for our use case. Assuming we do go ahead with implementing callbacks for asynchronous publisher events, what needs to be thought about/discussed before moving on to writing/accepting a patch? I have two ideas on how to implement this, but keep in mind that I'm rather new to OpenMAMA and market data in general. We could use a mamaTransportTopic event, and report the errors with a callback on the transport. The other option that I see, which Glenn hinted at, is that we create a structure similar to mamaMsgCallbacks, called mamaPublisherCallbacks; and a pair of new methods mamaPublisher_createWithCB and/or mamaPublisher_setCallbacks. That way we report the error on the specific publisher. What are your thoughts? Regards, Sam On 14-11-07 10:57 AM, Glenn McClements wrote: Hi Tom, Sam, Before you responded I was considering the use of an inbox as a solution to Sam’s query but I wasn’t overly keen on it. The reason being that being rejected by the broker felt very much like a middleware level event, not an application or market data event so it feels better to me that the error comes from the middleware as a callback. Also, responding with a MamaMsg to the sender means assuming the connection is up and alive, whereas a middleware ACL policy might actually prevent this from happening. It could be that you form the message on the client side in the bridge, but this itself feels incorrect as it’s translating from a middleware level even into a data message. (Also a minor point is that the MAMA_MSG_TYPE_SEC_STATUS relates to security status events on a exchange). This is not to say that a message response is incorrect for RMDS posts however, it’s just a different type of event. For Sam’s point I do prefer the idea of currently returning a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and the transport is currently the best/only place to do this. Going forward we could add callbacks to the MamaPublisher object to inform the client asynchronously of events like this, and MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED seems reasonable though we would need to consider how this would be used/interact with the higher level concept of market data entitlements. Another related concept is guaranteed or acknowledged messaging, which would generate other asynchronous event on a publishers. Like the original issue, this could be done at a high level with MamaMsgs being passed back to acknowledge delivery, but generally this is implemented at the middleware level so again a publisher callback would be better (with a handle to the original message as well for context). Glenn On 07/11/2014 14:20, "Tom Doust" <tom.doust@...> wrote: Hi In the Tick42 RMDS bridge we address this problem by allowing applications to use the sendFromInbox methods on a publisher and where applicable converting the an asynchronous response from the RMDS into a mama message which is delivered to the inbox. Although this is intended to support the RMDS message "post" publishing model that will ACK (or NAK) every message I think it will work for any middleware that has an asynchronous response mechanism; other publishing models on RMDS do not have a message reponse. The message sent to the inbox is arbitrary but it would make sense if other bridges that used this technique used the same message structure. The message we use is formed as follows mamaMsg_addI32(msg, "MdMsgType", 1, MAMA_MSG_TYPE_SEC_STATUS); mamaMsgStatus status = MAMA_MSG_STATUS_OK; if(nakCode != RSSL_NAKC_NONE) { status = RsslNakCode2MamaMsgStatus(nakCode); } mamaMsg_addI32(msg, "MdMsgStatus", 2, status); mamaMsg_addString(msg, "wSymbol",470, symbol_.c_str()); As you can see we just convert an rssl NAK code to an equivalent mama code, or use an OK status in the case of an ACK. These include states like MAMA_MSG_STATUS_NOT_ENTITLED and MAMA_MSG_STATUS_BAD_SYMBOL amongst a number of others. Obviously the set of code and the mappings are platform specific The one thing I think is missing here is a field containing a text string which could carry more information relevant to the client application. We would propose that everyone addressing the issue of processing asynchronous publishing responses takes this approach and that as a community we agree on a convention for the message content. Best Regards Tom Doust -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Sam Wilson Sent: 06 November 2014 7:45 PM To: openmama-dev@... Subject: [Openmama-dev] Asynchronous publisher events Hey all, We're looking for a way to notify a mama application asynchronously of errors while publishing, specifically when the published message was rejected by the message broker's ACL. Our API assumes the messages were accepted, and returns immediately. After some time you might get a connection-level event callback informing the application that its messages were rejected. I have two questions. First off, what is the best way to represent this situation in 2.3.1? We've been tossing around the idea of just tearing down the connection, giving the application a MAMA_TRANSPORT_DISCONNECT or a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and logging the error. Since the application or the environment is incorrectly configured, there's not much that can be done by the application anyways. Secondly, if openmama were to be extended in the future to support these kind of events, what would such extensions look like? Perhaps adding a MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED to the mamaTransportTopicEvent enumeration? Thanks, Sam _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev
More All Messages By This Member
Re: Asynchronous publisher events Gary Molloy <g.molloy@...> #1395 Hi Guys, I would like to propose a call this week (or next week) to kick off the discussion for the new mamaPublisher event handling feature. I was hoping to schedule a call either Thursday 11th, or Tuesday 16th of December @ 4pm UTC. I would appreciate it if you could let us know which date would suit most? Conference call details: UK Number: +44-28-90998719 US Number: +646-828-7349 Conference ID: 4741664 Thanks Gary Gary Molloy – SR Labs Adelaide Exchange \| 24-26 Adelaide Street \| Belfast \| BT2 8GD Tel: +44 28 9099 7580 Ext 3397 g.molloy@... toggle quoted message Show quoted text -----Original Message----- From: Gary Molloy Sent: 02 December 2014 17:44 To: Glenn McClements; Sam Wilson; Tom Doust; openmama-dev@... Subject: RE: [Openmama-dev] Asynchronous publisher events Hi Guys, I have created an appraisal / proposal document for the new publisher event handling feature on Google Docs which you can find here: https://docs.google.com/document/d/1bxy0Muxit4uc8Bz-y6CS3wFq5I7sJ0qAkA8-zr5vLec/edit?usp=sharing If you would like to review the document and we can then schedule a call so we can go over the proposal to discuss it in further detail. Thanks, Gary Gary Molloy SR.LABS Proven High Speed Electronic Trading Solutions g.molloy@... -----Original Message----- From: Glenn McClements Sent: 13 November 2014 12:15 To: Sam Wilson; Tom Doust; openmama-dev@... Cc: Gary Molloy Subject: Re: [Openmama-dev] Asynchronous publisher events Yeah I think something analogous to mamaMsgCallbacks (which should have been called mamaSubscriberCallbacks in hindsight) for the mamaPublisher is the nicest fit here. The advantage of doing it this was is that the set of callbacks can be optional and extensible. One gotcha though is the question of which thread/queue the callback should be fired on? Currently the mamaPublisher doesn’t have a queue associated with it so we’ll probably need to add this as well. For next steps I propose we collate the requirements for publisher events and put together a proposal/appraisal, then we can figure out who wants to implement and test it. Possible events are: - undeliverable (with a reason/status) - delivered - …. anything else? Gary on our side will pull these together and we can set up a call to discuss. Cheers, Glenn On 11/11/2014 16:45, "Sam Wilson" <Sam.Wilson@...> wrote: Hey Glenn, Tom, I have to agree with Glenn here. I don't think these kind of events belong alongside regular messages, for a couple of reasons. For one, simply enough, they aren't messages. They don't originate from a publisher, they wouldn't travel along the wire, and they probably would have to be synthesized in the middleware bridge. Secondly, Tom's solution only works for sendFromInbox. The callback mechanism is much more general, and would work for non-inbox messages as well, which is important for our use case. Assuming we do go ahead with implementing callbacks for asynchronous publisher events, what needs to be thought about/discussed before moving on to writing/accepting a patch? I have two ideas on how to implement this, but keep in mind that I'm rather new to OpenMAMA and market data in general. We could use a mamaTransportTopic event, and report the errors with a callback on the transport. The other option that I see, which Glenn hinted at, is that we create a structure similar to mamaMsgCallbacks, called mamaPublisherCallbacks; and a pair of new methods mamaPublisher_createWithCB and/or mamaPublisher_setCallbacks. That way we report the error on the specific publisher. What are your thoughts? Regards, Sam On 14-11-07 10:57 AM, Glenn McClements wrote: Hi Tom, Sam, Before you responded I was considering the use of an inbox as a solution to Sam’s query but I wasn’t overly keen on it. The reason being that being rejected by the broker felt very much like a middleware level event, not an application or market data event so it feels better to me that the error comes from the middleware as a callback. Also, responding with a MamaMsg to the sender means assuming the connection is up and alive, whereas a middleware ACL policy might actually prevent this from happening. It could be that you form the message on the client side in the bridge, but this itself feels incorrect as it’s translating from a middleware level even into a data message. (Also a minor point is that the MAMA_MSG_TYPE_SEC_STATUS relates to security status events on a exchange). This is not to say that a message response is incorrect for RMDS posts however, it’s just a different type of event. For Sam’s point I do prefer the idea of currently returning a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and the transport is currently the best/only place to do this. Going forward we could add callbacks to the MamaPublisher object to inform the client asynchronously of events like this, and MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED seems reasonable though we would need to consider how this would be used/interact with the higher level concept of market data entitlements. Another related concept is guaranteed or acknowledged messaging, which would generate other asynchronous event on a publishers. Like the original issue, this could be done at a high level with MamaMsgs being passed back to acknowledge delivery, but generally this is implemented at the middleware level so again a publisher callback would be better (with a handle to the original message as well for context). Glenn On 07/11/2014 14:20, "Tom Doust" <tom.doust@...> wrote: Hi In the Tick42 RMDS bridge we address this problem by allowing applications to use the sendFromInbox methods on a publisher and where applicable converting the an asynchronous response from the RMDS into a mama message which is delivered to the inbox. Although this is intended to support the RMDS message "post" publishing model that will ACK (or NAK) every message I think it will work for any middleware that has an asynchronous response mechanism; other publishing models on RMDS do not have a message reponse. The message sent to the inbox is arbitrary but it would make sense if other bridges that used this technique used the same message structure. The message we use is formed as follows mamaMsg_addI32(msg, "MdMsgType", 1, MAMA_MSG_TYPE_SEC_STATUS); mamaMsgStatus status = MAMA_MSG_STATUS_OK; if(nakCode != RSSL_NAKC_NONE) { status = RsslNakCode2MamaMsgStatus(nakCode); } mamaMsg_addI32(msg, "MdMsgStatus", 2, status); mamaMsg_addString(msg, "wSymbol",470, symbol_.c_str()); As you can see we just convert an rssl NAK code to an equivalent mama code, or use an OK status in the case of an ACK. These include states like MAMA_MSG_STATUS_NOT_ENTITLED and MAMA_MSG_STATUS_BAD_SYMBOL amongst a number of others. Obviously the set of code and the mappings are platform specific The one thing I think is missing here is a field containing a text string which could carry more information relevant to the client application. We would propose that everyone addressing the issue of processing asynchronous publishing responses takes this approach and that as a community we agree on a convention for the message content. Best Regards Tom Doust -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Sam Wilson Sent: 06 November 2014 7:45 PM To: openmama-dev@... Subject: [Openmama-dev] Asynchronous publisher events Hey all, We're looking for a way to notify a mama application asynchronously of errors while publishing, specifically when the published message was rejected by the message broker's ACL. Our API assumes the messages were accepted, and returns immediately. After some time you might get a connection-level event callback informing the application that its messages were rejected. I have two questions. First off, what is the best way to represent this situation in 2.3.1? We've been tossing around the idea of just tearing down the connection, giving the application a MAMA_TRANSPORT_DISCONNECT or a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and logging the error. Since the application or the environment is incorrectly configured, there's not much that can be done by the application anyways. Secondly, if openmama were to be extended in the future to support these kind of events, what would such extensions look like? Perhaps adding a MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED to the mamaTransportTopicEvent enumeration? Thanks, Sam _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev
More All Messages By This Member
Re: Asynchronous publisher events Gary Molloy <g.molloy@...> #1393 Hi Guys, I have created an appraisal / proposal document for the new publisher event handling feature on Google Docs which you can find here: https://docs.google.com/document/d/1bxy0Muxit4uc8Bz-y6CS3wFq5I7sJ0qAkA8-zr5vLec/edit?usp=sharing If you would like to review the document and we can then schedule a call so we can go over the proposal to discuss it in further detail. Thanks, Gary Gary Molloy SR.LABS Proven High Speed Electronic Trading Solutions g.molloy@... toggle quoted message Show quoted text -----Original Message----- From: Glenn McClements Sent: 13 November 2014 12:15 To: Sam Wilson; Tom Doust; openmama-dev@... Cc: Gary Molloy Subject: Re: [Openmama-dev] Asynchronous publisher events Yeah I think something analogous to mamaMsgCallbacks (which should have been called mamaSubscriberCallbacks in hindsight) for the mamaPublisher is the nicest fit here. The advantage of doing it this was is that the set of callbacks can be optional and extensible. One gotcha though is the question of which thread/queue the callback should be fired on? Currently the mamaPublisher doesn’t have a queue associated with it so we’ll probably need to add this as well. For next steps I propose we collate the requirements for publisher events and put together a proposal/appraisal, then we can figure out who wants to implement and test it. Possible events are: - undeliverable (with a reason/status) - delivered - …. anything else? Gary on our side will pull these together and we can set up a call to discuss. Cheers, Glenn On 11/11/2014 16:45, "Sam Wilson" <Sam.Wilson@...> wrote: Hey Glenn, Tom, I have to agree with Glenn here. I don't think these kind of events belong alongside regular messages, for a couple of reasons. For one, simply enough, they aren't messages. They don't originate from a publisher, they wouldn't travel along the wire, and they probably would have to be synthesized in the middleware bridge. Secondly, Tom's solution only works for sendFromInbox. The callback mechanism is much more general, and would work for non-inbox messages as well, which is important for our use case. Assuming we do go ahead with implementing callbacks for asynchronous publisher events, what needs to be thought about/discussed before moving on to writing/accepting a patch? I have two ideas on how to implement this, but keep in mind that I'm rather new to OpenMAMA and market data in general. We could use a mamaTransportTopic event, and report the errors with a callback on the transport. The other option that I see, which Glenn hinted at, is that we create a structure similar to mamaMsgCallbacks, called mamaPublisherCallbacks; and a pair of new methods mamaPublisher_createWithCB and/or mamaPublisher_setCallbacks. That way we report the error on the specific publisher. What are your thoughts? Regards, Sam On 14-11-07 10:57 AM, Glenn McClements wrote: Hi Tom, Sam, Before you responded I was considering the use of an inbox as a solution to Sam’s query but I wasn’t overly keen on it. The reason being that being rejected by the broker felt very much like a middleware level event, not an application or market data event so it feels better to me that the error comes from the middleware as a callback. Also, responding with a MamaMsg to the sender means assuming the connection is up and alive, whereas a middleware ACL policy might actually prevent this from happening. It could be that you form the message on the client side in the bridge, but this itself feels incorrect as it’s translating from a middleware level even into a data message. (Also a minor point is that the MAMA_MSG_TYPE_SEC_STATUS relates to security status events on a exchange). This is not to say that a message response is incorrect for RMDS posts however, it’s just a different type of event. For Sam’s point I do prefer the idea of currently returning a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and the transport is currently the best/only place to do this. Going forward we could add callbacks to the MamaPublisher object to inform the client asynchronously of events like this, and MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED seems reasonable though we would need to consider how this would be used/interact with the higher level concept of market data entitlements. Another related concept is guaranteed or acknowledged messaging, which would generate other asynchronous event on a publishers. Like the original issue, this could be done at a high level with MamaMsgs being passed back to acknowledge delivery, but generally this is implemented at the middleware level so again a publisher callback would be better (with a handle to the original message as well for context). Glenn On 07/11/2014 14:20, "Tom Doust" <tom.doust@...> wrote: Hi In the Tick42 RMDS bridge we address this problem by allowing applications to use the sendFromInbox methods on a publisher and where applicable converting the an asynchronous response from the RMDS into a mama message which is delivered to the inbox. Although this is intended to support the RMDS message "post" publishing model that will ACK (or NAK) every message I think it will work for any middleware that has an asynchronous response mechanism; other publishing models on RMDS do not have a message reponse. The message sent to the inbox is arbitrary but it would make sense if other bridges that used this technique used the same message structure. The message we use is formed as follows mamaMsg_addI32(msg, "MdMsgType", 1, MAMA_MSG_TYPE_SEC_STATUS); mamaMsgStatus status = MAMA_MSG_STATUS_OK; if(nakCode != RSSL_NAKC_NONE) { status = RsslNakCode2MamaMsgStatus(nakCode); } mamaMsg_addI32(msg, "MdMsgStatus", 2, status); mamaMsg_addString(msg, "wSymbol",470, symbol_.c_str()); As you can see we just convert an rssl NAK code to an equivalent mama code, or use an OK status in the case of an ACK. These include states like MAMA_MSG_STATUS_NOT_ENTITLED and MAMA_MSG_STATUS_BAD_SYMBOL amongst a number of others. Obviously the set of code and the mappings are platform specific The one thing I think is missing here is a field containing a text string which could carry more information relevant to the client application. We would propose that everyone addressing the issue of processing asynchronous publishing responses takes this approach and that as a community we agree on a convention for the message content. Best Regards Tom Doust -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Sam Wilson Sent: 06 November 2014 7:45 PM To: openmama-dev@... Subject: [Openmama-dev] Asynchronous publisher events Hey all, We're looking for a way to notify a mama application asynchronously of errors while publishing, specifically when the published message was rejected by the message broker's ACL. Our API assumes the messages were accepted, and returns immediately. After some time you might get a connection-level event callback informing the application that its messages were rejected. I have two questions. First off, what is the best way to represent this situation in 2.3.1? We've been tossing around the idea of just tearing down the connection, giving the application a MAMA_TRANSPORT_DISCONNECT or a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and logging the error. Since the application or the environment is incorrectly configured, there's not much that can be done by the application anyways. Secondly, if openmama were to be extended in the future to support these kind of events, what would such extensions look like? Perhaps adding a MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED to the mamaTransportTopicEvent enumeration? Thanks, Sam _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev
More All Messages By This Member
Re: Enforcing field type on publish Gary Molloy <g.molloy@...> #1392 Hi Reed, Thanks for getting back to us. I think, as you say, the goal will be to have this feature as a plugin and not as an #ifdef, but we would be very interested in seeing your current implementation with an #ifdef patch to keep the discussion going on this. Thanks, Gary Gary Molloy SR.LABS Proven High Speed Electronic Trading Solutions g.molloy@... toggle quoted message Show quoted text From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Alpert, Reed Sent: 18 November 2014 15:10 To: Alpert, Reed; Damian Maguire Cc: Openmama-dev@...; Glenn McClements; Frank Quinn Subject: Re: [Openmama-dev] Enforcing field type on publish Hi Damian, I have our pre-publish check code ready, but it is not pluggable. I’d like to submit it as a patch using #ifdef like the entitlements code. That way I don’t have to maintain it as OM evolves, and it won’t affect other OM developers. When the pluggable/hook system is ready I can port it over. There are 2 new files: mama/c_cpp/src/c/publisherfieldcheck.c mama/c_cpp/src/c/publisherfieldcheck.h and changes to: mama/c_cpp/src/c/Sconscript mama/c_cpp/src/c/transport.c mama/c_cpp/src/c/mama/transport.h mama/c_cpp/src/c/publisher.c Please let me know if an #ifdef patch is OK for this. I can submit the patch if you want to see what it involves. Thanks, Reed. Reed Alpert \| Corporate & Investment Bank \| Market Data Services \| J.P. Morgan \| 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 \| T: 718.242.5198 \| M: 917.414.4613 \| reed.alpert@... Alternate Contact: CIB PIM Trading Technology Solutions NA \| CIB_PIM_Trading_Technology_Solutions_NA@... From: Alpert, Reed Sent: Monday, September 22, 2014 6:17 PM To: 'Damian Maguire' Cc: Glenn McClements; Benjamin Taieb; Frank Quinn; fquinn.ni@...; Openmama-dev@... Subject: RE: [Openmama-dev] Enforcing field type on publish Hi Damian, Yes, having a pluggable module that handles some pre/post events makes sense. Our handler for this now is very JPM-specific, and I don’t think it would make sense to try and standardize that (checks field types, ensures there is a msgType, and adds some audit fields to track who/when/where published the msg). To get a dictionary I put a hook in at the end of the transport create, since that is the first time we have enough info to check the config in mama.properties and then subscribe to the dictionary. If that is OK then the dictionary is stashed in the transport data, and the pre-publish looks at that to see if it should apply the checks. Begin able to load a shared obj via the library manager would work great. For the pre/post events it seems like a lot to have a general list like transport create and publish and maybe bridge create, etc. I’m not sure if that is overkill to fully fill out the list of pre/post events across many objects. To store library data in the main objects’ data (e.g., bridge, transport, publisher, etc) maybe a user closure can be added. Of course that assumes that a single pre/post library is installed. I’ll keep adding things as our development continue. Thanks, Reed. Reed Alpert \| Corporate & Investment Bank \| Market Data Services \| J.P. Morgan \| 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 \| T: 718.242.5198 \| M: 917.414.4613 \| reed.alpert@... Alternate Contact: CIB PIM Trading Technology Solutions NA \| CIB_PIM_Trading_Technology_Solutions_NA@... From: Damian Maguire [mailto:damian@...] Sent: Friday, September 19, 2014 11:53 AM To: Alpert, Reed Cc: Glenn McClements; Benjamin Taieb; Frank Quinn; fquinn.ni@...; Openmama-dev@... Subject: Re: [Openmama-dev] Enforcing field type on publish Hey Reed, This all sounds really interesting, particularly the fact that you have already identified multiple use cases for publish side hooks. A lot of it actually ties very nicely with some of the internal discussions we've been having around a plugin architecture for OpenMAMA as well, which in turn sit quite nicely with the library manager work which is also ongoing. At present myself and the team here are pretty solidly booked with a few internal projects (which should provide some real benefits to OpenMAMA when they're complete), so it'll be a few weeks before we can sit down and tackle this properly. However, I'd be very keen to get some full community engagement in discussing this, and work to put together a design that covers the requirements of the majority of potential use cases. With that in mind, if you guys can think of anywhere else where you believe hooks would be useful, or if you can think of any specific requirements for your current ideas, feel free to send them over and we'll start collating everything. Thanks, Damian On Wed, Sep 17, 2014 at 8:03 PM, Alpert, Reed <reed.alpert@...> wrote: Hi, I agree, the callback to a module that can do publish checking/etc is good. I prototyped a checking module, and it works, although loading the dictionary on 1^st publish has a small delay, and also required that the publish not be from a mama timer callback, since that deadlocked either on sending the subscription on the throttle queue, or getting the dict back. We actually have more requirements for publish that are very specific to our env (adding system audit fields to all publishes to track when/where the data comes from). Having a callback we can engage would allow us to do this w/o any other users needing to have our stuff in the way. So loading our module at runtime and calling it would work quite well. Thanks, Reed. Reed Alpert \| Corporate & Investment Bank \| Market Data Services \| J.P. Morgan \| 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 \| T: 718.242.5198 \| M: 917.414.4613 \| reed.alpert@... Alternate Contact: CIB PIM Trading Technology Solutions NA \| CIB_PIM_Trading_Technology_Solutions_NA@... From: Glenn McClements [mailto:gmcclements@...] Sent: Thursday, September 11, 2014 5:20 AM To: Alpert, Reed; Benjamin Taieb; Frank Quinn; fquinn.ni@... Cc: Openmama-dev@... Subject: Re: [Openmama-dev] Enforcing field type on publish Hi Alpert, I like #2 more, but I still want to keep the more data model/market data/custom logic away from the core messaging part of OpenMAMA. What I’m thinking of is a hook/callback to a programmer defined function that would take the publisher, transport, message, a user defined closure etc that would return a status to indicate if the message should be published or not. The dictionary etc could be obtained at initialisation time through a separate init function. This dovetails with some of the other pieces of work going on: we now have a framework for loading bridges dynamically, so the functionality could be in a library loaded automatically at startup the hook above looks very much like publisher side entitlements (pass a publisher, message and expect a yes or no back) so we do I still need to work though this a bit more but let me know what you think. Glenn From: <Alpert>, Reed <reed.alpert@...> Date: Wednesday, 10 September 2014 19:13 To: Benjamin Taieb <benjamin.taieb@...>, Frank Quinn <fquinn@...>, "fquinn.ni@..." <fquinn.ni@...>, Glenn McClements <gmcclements@...> Cc: "Openmama-dev@..." <Openmama-dev@...> Subject: RE: [Openmama-dev] Enforcing field type on publish Hi, Yes I agree with Ben’s comments. Here are some caveats: 1. This is a feature that is requested by our market data infrastructure team to prevent app dev teams from accidentally publishing incorrect field types. We don’t believe that a voluntary program will work, which is why the FieldCache as a location is not preferred. Apps won’t use the FieldCache if they don’t need it, or are using an Excel plugin for publishing (quite common). 2. This is a configured feature, and only for publication. 3. This is not for subscription, that is, a subscriber can use any of the getXX() methods that work. There seems to be general agreement that the OM layer is much better place for this feature than bridges themselves. The two best locations seem to be: 1. In mamaMsg, in the addXX and updateXX methods, probably with a macro like CHECK_MODIFY() that calls out to another module to do the work. a. This is good since it keep the message construction in mamaMsg module. b. This is good since it give the programmer the quickest feedback on an error condition. c. This is bad since it puts some performance hit on construction of every message, even on subscribe. d. This is bad since it makes the code active for creating subscribe messages too. e. This is bad since it puts more code in more places for this feature. 2. In publisher.send() that checks a config var and then calls out to another module to do the work. a. This is good since it has minimal code changes, and minimal performance effect. b. This is good since it only affects publish messages. c. This is bad since it gives delayed failure, and there may be multiple fields that fail. i. Probably logging each failed field and returning an error code will work well to help developers fix problems. I think the publisher.send() method is the best place, given the above. But figuring out which dictionary to use for the checking seems to be the most difficult part. Adding this feature as a config option may require also adding some optional config info about how to get a dictionary for a specific bridge, e.g., the source name. Once the dictionary loading/mapping is solved then the rest seems fairly straightforward, except for the determination of allowed/denied conversions as configurable policy, but starting with 3 simple ones seems best: 1. No checking (default) 2. Type must match dictionary. 3. Type must match dictionary, with a small number of allowed conversions that do not incur data loss (e.g., F32 -> F64). Thanks, Reed. This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired. This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities. _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.
More All Messages By This Member
Re: OpenMAMA RPM Release Scripts Alpert, Reed <reed.alpert@...> #1391 Hi Damian, Just a note: the release subdir is also used by Vlsual Studio for its output, so there will be the rpm scripts and lib/dll/pdb files from the builds. Our Windows git looks like this after running both Visual Studio and scons builds: Thanks, Reed. Reed Alpert \| Corporate & Investment Bank \| Market Data Services \| J.P. Morgan \| 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 \| T: 718.242.5198 \| M: 917.414.4613 \| reed.alpert@... Alternate Contact: CIB PIM Trading Technology Solutions NA \| CIB_PIM_Trading_Technology_Solutions_NA@... toggle quoted message Show quoted text From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Damian Maguire Sent: Wednesday, November 12, 2014 9:31 AM To: openmama-dev@... Subject: [Openmama-dev] OpenMAMA RPM Release Scripts Folks, Some of you may have noticed a few commits which have just landed in next which relate to the OpenMAMA RPM builds. These are tools which I've been using internally to help facilitate the OpenMAMA release process by automating some of the steps involved. At present they are pretty rough around the edges, however after talking to a few people about them I believe there is some value in having them out there for others to review/enhance. The primary one of interest to most people is likely to be the openmama.spec file, which resides under the 'release' directory. This is the specification file used by rpmbuild to generate the base RPM and SRPMs, which can then used to generate other RPMs for alternative platforms. If you clone down next (and assuming you have met the base requirements - running RHEL6 or derivative, have all the OpenMAMA pre-reqs, have rpmbuild and mock installed etc), you should be able to perform a full build yourself by simply entering the release directory and executing: $ ./openmama-rpm.sh As I say, it's all a little rough, and still quite heavily tailored to my own build environment, but I would appreciate any feedback/bug fixes/issues etc. Cheers, D This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.
More All Messages By This Member
Re: Enforcing field type on publish Alpert, Reed <reed.alpert@...> #1390 Hi Damian, I have our pre-publish check code ready, but it is not pluggable. I’d like to submit it as a patch using #ifdef like the entitlements code. That way I don’t have to maintain it as OM evolves, and it won’t affect other OM developers. When the pluggable/hook system is ready I can port it over. There are 2 new files: mama/c_cpp/src/c/publisherfieldcheck.c mama/c_cpp/src/c/publisherfieldcheck.h and changes to: mama/c_cpp/src/c/Sconscript mama/c_cpp/src/c/transport.c mama/c_cpp/src/c/mama/transport.h mama/c_cpp/src/c/publisher.c Please let me know if an #ifdef patch is OK for this. I can submit the patch if you want to see what it involves. Thanks, Reed. Reed Alpert \| Corporate & Investment Bank \| Market Data Services \| J.P. Morgan \| 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 \| T: 718.242.5198 \| M: 917.414.4613 \| reed.alpert@... Alternate Contact: CIB PIM Trading Technology Solutions NA \| CIB_PIM_Trading_Technology_Solutions_NA@... toggle quoted message Show quoted text From: Alpert, Reed Sent: Monday, September 22, 2014 6:17 PM To: 'Damian Maguire' Cc: Glenn McClements; Benjamin Taieb; Frank Quinn; fquinn.ni@...; Openmama-dev@... Subject: RE: [Openmama-dev] Enforcing field type on publish Hi Damian, Yes, having a pluggable module that handles some pre/post events makes sense. Our handler for this now is very JPM-specific, and I don’t think it would make sense to try and standardize that (checks field types, ensures there is a msgType, and adds some audit fields to track who/when/where published the msg). To get a dictionary I put a hook in at the end of the transport create, since that is the first time we have enough info to check the config in mama.properties and then subscribe to the dictionary. If that is OK then the dictionary is stashed in the transport data, and the pre-publish looks at that to see if it should apply the checks. Begin able to load a shared obj via the library manager would work great. For the pre/post events it seems like a lot to have a general list like transport create and publish and maybe bridge create, etc. I’m not sure if that is overkill to fully fill out the list of pre/post events across many objects. To store library data in the main objects’ data (e.g., bridge, transport, publisher, etc) maybe a user closure can be added. Of course that assumes that a single pre/post library is installed. I’ll keep adding things as our development continue. Thanks, Reed. Reed Alpert \| Corporate & Investment Bank \| Market Data Services \| J.P. Morgan \| 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 \| T: 718.242.5198 \| M: 917.414.4613 \| reed.alpert@... Alternate Contact: CIB PIM Trading Technology Solutions NA \| CIB_PIM_Trading_Technology_Solutions_NA@... From: Damian Maguire [mailto:damian@...] Sent: Friday, September 19, 2014 11:53 AM To: Alpert, Reed Cc: Glenn McClements; Benjamin Taieb; Frank Quinn; fquinn.ni@...; Openmama-dev@... Subject: Re: [Openmama-dev] Enforcing field type on publish Hey Reed, This all sounds really interesting, particularly the fact that you have already identified multiple use cases for publish side hooks. A lot of it actually ties very nicely with some of the internal discussions we've been having around a plugin architecture for OpenMAMA as well, which in turn sit quite nicely with the library manager work which is also ongoing. At present myself and the team here are pretty solidly booked with a few internal projects (which should provide some real benefits to OpenMAMA when they're complete), so it'll be a few weeks before we can sit down and tackle this properly. However, I'd be very keen to get some full community engagement in discussing this, and work to put together a design that covers the requirements of the majority of potential use cases. With that in mind, if you guys can think of anywhere else where you believe hooks would be useful, or if you can think of any specific requirements for your current ideas, feel free to send them over and we'll start collating everything. Thanks, Damian On Wed, Sep 17, 2014 at 8:03 PM, Alpert, Reed <reed.alpert@...> wrote: Hi, I agree, the callback to a module that can do publish checking/etc is good. I prototyped a checking module, and it works, although loading the dictionary on 1^st publish has a small delay, and also required that the publish not be from a mama timer callback, since that deadlocked either on sending the subscription on the throttle queue, or getting the dict back. We actually have more requirements for publish that are very specific to our env (adding system audit fields to all publishes to track when/where the data comes from). Having a callback we can engage would allow us to do this w/o any other users needing to have our stuff in the way. So loading our module at runtime and calling it would work quite well. Thanks, Reed. Reed Alpert \| Corporate & Investment Bank \| Market Data Services \| J.P. Morgan \| 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 \| T: 718.242.5198 \| M: 917.414.4613 \| reed.alpert@... Alternate Contact: CIB PIM Trading Technology Solutions NA \| CIB_PIM_Trading_Technology_Solutions_NA@... From: Glenn McClements [mailto:gmcclements@...] Sent: Thursday, September 11, 2014 5:20 AM To: Alpert, Reed; Benjamin Taieb; Frank Quinn; fquinn.ni@... Cc: Openmama-dev@... Subject: Re: [Openmama-dev] Enforcing field type on publish Hi Alpert, I like #2 more, but I still want to keep the more data model/market data/custom logic away from the core messaging part of OpenMAMA. What I’m thinking of is a hook/callback to a programmer defined function that would take the publisher, transport, message, a user defined closure etc that would return a status to indicate if the message should be published or not. The dictionary etc could be obtained at initialisation time through a separate init function. This dovetails with some of the other pieces of work going on: we now have a framework for loading bridges dynamically, so the functionality could be in a library loaded automatically at startup the hook above looks very much like publisher side entitlements (pass a publisher, message and expect a yes or no back) so we do I still need to work though this a bit more but let me know what you think. Glenn From: <Alpert>, Reed <reed.alpert@...> Date: Wednesday, 10 September 2014 19:13 To: Benjamin Taieb <benjamin.taieb@...>, Frank Quinn <fquinn@...>, "fquinn.ni@..." <fquinn.ni@...>, Glenn McClements <gmcclements@...> Cc: "Openmama-dev@..." <Openmama-dev@...> Subject: RE: [Openmama-dev] Enforcing field type on publish Hi, Yes I agree with Ben’s comments. Here are some caveats: 1. This is a feature that is requested by our market data infrastructure team to prevent app dev teams from accidentally publishing incorrect field types. We don’t believe that a voluntary program will work, which is why the FieldCache as a location is not preferred. Apps won’t use the FieldCache if they don’t need it, or are using an Excel plugin for publishing (quite common). 2. This is a configured feature, and only for publication. 3. This is not for subscription, that is, a subscriber can use any of the getXX() methods that work. There seems to be general agreement that the OM layer is much better place for this feature than bridges themselves. The two best locations seem to be: 1. In mamaMsg, in the addXX and updateXX methods, probably with a macro like CHECK_MODIFY() that calls out to another module to do the work. a. This is good since it keep the message construction in mamaMsg module. b. This is good since it give the programmer the quickest feedback on an error condition. c. This is bad since it puts some performance hit on construction of every message, even on subscribe. d. This is bad since it makes the code active for creating subscribe messages too. e. This is bad since it puts more code in more places for this feature. 2. In publisher.send() that checks a config var and then calls out to another module to do the work. a. This is good since it has minimal code changes, and minimal performance effect. b. This is good since it only affects publish messages. c. This is bad since it gives delayed failure, and there may be multiple fields that fail. i. Probably logging each failed field and returning an error code will work well to help developers fix problems. I think the publisher.send() method is the best place, given the above. But figuring out which dictionary to use for the checking seems to be the most difficult part. Adding this feature as a config option may require also adding some optional config info about how to get a dictionary for a specific bridge, e.g., the source name. Once the dictionary loading/mapping is solved then the rest seems fairly straightforward, except for the determination of allowed/denied conversions as configurable policy, but starting with 3 simple ones seems best: 1. No checking (default) 2. Type must match dictionary. 3. Type must match dictionary, with a small number of allowed conversions that do not incur data loss (e.g., F32 -> F64). Thanks, Reed. This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired. This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities. _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.
More All Messages By This Member
Re: Asynchronous publisher events Glenn McClements <g.mcclements@...> #1389 Yeah I think something analogous to mamaMsgCallbacks (which should have been called mamaSubscriberCallbacks in hindsight) for the mamaPublisher is the nicest fit here. The advantage of doing it this was is that the set of callbacks can be optional and extensible. One gotcha though is the question of which thread/queue the callback should be fired on? Currently the mamaPublisher doesn’t have a queue associated with it so we’ll probably need to add this as well. For next steps I propose we collate the requirements for publisher events and put together a proposal/appraisal, then we can figure out who wants to implement and test it. Possible events are: - undeliverable (with a reason/status) - delivered - …. anything else? Gary on our side will pull these together and we can set up a call to discuss. Cheers, Glenn toggle quoted message Show quoted text On 11/11/2014 16:45, "Sam Wilson" <Sam.Wilson@...> wrote: Hey Glenn, Tom, I have to agree with Glenn here. I don't think these kind of events belong alongside regular messages, for a couple of reasons. For one, simply enough, they aren't messages. They don't originate from a publisher, they wouldn't travel along the wire, and they probably would have to be synthesized in the middleware bridge. Secondly, Tom's solution only works for sendFromInbox. The callback mechanism is much more general, and would work for non-inbox messages as well, which is important for our use case. Assuming we do go ahead with implementing callbacks for asynchronous publisher events, what needs to be thought about/discussed before moving on to writing/accepting a patch? I have two ideas on how to implement this, but keep in mind that I'm rather new to OpenMAMA and market data in general. We could use a mamaTransportTopic event, and report the errors with a callback on the transport. The other option that I see, which Glenn hinted at, is that we create a structure similar to mamaMsgCallbacks, called mamaPublisherCallbacks; and a pair of new methods mamaPublisher_createWithCB and/or mamaPublisher_setCallbacks. That way we report the error on the specific publisher. What are your thoughts? Regards, Sam On 14-11-07 10:57 AM, Glenn McClements wrote: Hi Tom, Sam, Before you responded I was considering the use of an inbox as a solution to Sam’s query but I wasn’t overly keen on it. The reason being that being rejected by the broker felt very much like a middleware level event, not an application or market data event so it feels better to me that the error comes from the middleware as a callback. Also, responding with a MamaMsg to the sender means assuming the connection is up and alive, whereas a middleware ACL policy might actually prevent this from happening. It could be that you form the message on the client side in the bridge, but this itself feels incorrect as it’s translating from a middleware level even into a data message. (Also a minor point is that the MAMA_MSG_TYPE_SEC_STATUS relates to security status events on a exchange). This is not to say that a message response is incorrect for RMDS posts however, it’s just a different type of event. For Sam’s point I do prefer the idea of currently returning a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and the transport is currently the best/only place to do this. Going forward we could add callbacks to the MamaPublisher object to inform the client asynchronously of events like this, and MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED seems reasonable though we would need to consider how this would be used/interact with the higher level concept of market data entitlements. Another related concept is guaranteed or acknowledged messaging, which would generate other asynchronous event on a publishers. Like the original issue, this could be done at a high level with MamaMsgs being passed back to acknowledge delivery, but generally this is implemented at the middleware level so again a publisher callback would be better (with a handle to the original message as well for context). Glenn On 07/11/2014 14:20, "Tom Doust" <tom.doust@...> wrote: Hi In the Tick42 RMDS bridge we address this problem by allowing applications to use the sendFromInbox methods on a publisher and where applicable converting the an asynchronous response from the RMDS into a mama message which is delivered to the inbox. Although this is intended to support the RMDS message "post" publishing model that will ACK (or NAK) every message I think it will work for any middleware that has an asynchronous response mechanism; other publishing models on RMDS do not have a message reponse. The message sent to the inbox is arbitrary but it would make sense if other bridges that used this technique used the same message structure. The message we use is formed as follows mamaMsg_addI32(msg, "MdMsgType", 1, MAMA_MSG_TYPE_SEC_STATUS); mamaMsgStatus status = MAMA_MSG_STATUS_OK; if(nakCode != RSSL_NAKC_NONE) { status = RsslNakCode2MamaMsgStatus(nakCode); } mamaMsg_addI32(msg, "MdMsgStatus", 2, status); mamaMsg_addString(msg, "wSymbol",470, symbol_.c_str()); As you can see we just convert an rssl NAK code to an equivalent mama code, or use an OK status in the case of an ACK. These include states like MAMA_MSG_STATUS_NOT_ENTITLED and MAMA_MSG_STATUS_BAD_SYMBOL amongst a number of others. Obviously the set of code and the mappings are platform specific The one thing I think is missing here is a field containing a text string which could carry more information relevant to the client application. We would propose that everyone addressing the issue of processing asynchronous publishing responses takes this approach and that as a community we agree on a convention for the message content. Best Regards Tom Doust -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Sam Wilson Sent: 06 November 2014 7:45 PM To: openmama-dev@... Subject: [Openmama-dev] Asynchronous publisher events Hey all, We're looking for a way to notify a mama application asynchronously of errors while publishing, specifically when the published message was rejected by the message broker's ACL. Our API assumes the messages were accepted, and returns immediately. After some time you might get a connection-level event callback informing the application that its messages were rejected. I have two questions. First off, what is the best way to represent this situation in 2.3.1? We've been tossing around the idea of just tearing down the connection, giving the application a MAMA_TRANSPORT_DISCONNECT or a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and logging the error. Since the application or the environment is incorrectly configured, there's not much that can be done by the application anyways. Secondly, if openmama were to be extended in the future to support these kind of events, what would such extensions look like? Perhaps adding a MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED to the mamaTransportTopicEvent enumeration? Thanks, Sam _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev
More
OpenMAMA RPM Release Scripts Damian Maguire #1388 Folks, Some of you may have noticed a few commits which have just landed in next which relate to the OpenMAMA RPM builds. These are tools which I've been using internally to help facilitate the OpenMAMA release process by automating some of the steps involved. At present they are pretty rough around the edges, however after talking to a few people about them I believe there is some value in having them out there for others to review/enhance. The primary one of interest to most people is likely to be the openmama.spec file, which resides under the 'release' directory. This is the specification file used by rpmbuild to generate the base RPM and SRPMs, which can then used to generate other RPMs for alternative platforms. If you clone down next (and assuming you have met the base requirements - running RHEL6 or derivative, have all the OpenMAMA pre-reqs, have rpmbuild and mock installed etc), you should be able to perform a full build yourself by simply entering the release directory and executing: $ ./openmama-rpm.sh As I say, it's all a little rough, and still quite heavily tailored to my own build environment, but I would appreciate any feedback/bug fixes/issues etc. Cheers, D
More All Messages By This Member
Re: Asynchronous publisher events Tom Doust #1387 Hi Sam (and Glenn) Its interesting to hear the Solace perspective on this. In the RMDS case we are dealing with real messages, they come from the intended destination of the published message (or some routing/authentication element encountered en route to the destination) and they do come in on the wire. And they are associated with a publisher. Almost completely complementary to the case you describe I don’t really care how we deliver this kind of notification to the client. I agree though that callbacks are (in some respects) easier to implement. If publisher call-backs are added to the api I will modify our bridge to give client apps the option of using them. In the meantime inbox messages work just fine. I think one thing that this issue demonstrates is that it is important to look at the behaviour of a range of platforms as part of defining interfaces. Best regards Tom toggle quoted message Show quoted text -----Original Message----- From: Sam Wilson [mailto:Sam.Wilson@...] Sent: 11 November 2014 4:46 PM To: Glenn McClements; Tom Doust; openmama-dev@... Subject: Re: [Openmama-dev] Asynchronous publisher events Hey Glenn, Tom, I have to agree with Glenn here. I don't think these kind of events belong alongside regular messages, for a couple of reasons. For one, simply enough, they aren't messages. They don't originate from a publisher, they wouldn't travel along the wire, and they probably would have to be synthesized in the middleware bridge. Secondly, Tom's solution only works for sendFromInbox. The callback mechanism is much more general, and would work for non-inbox messages as well, which is important for our use case. Assuming we do go ahead with implementing callbacks for asynchronous publisher events, what needs to be thought about/discussed before moving on to writing/accepting a patch? I have two ideas on how to implement this, but keep in mind that I'm rather new to OpenMAMA and market data in general. We could use a mamaTransportTopic event, and report the errors with a callback on the transport. The other option that I see, which Glenn hinted at, is that we create a structure similar to mamaMsgCallbacks, called mamaPublisherCallbacks; and a pair of new methods mamaPublisher_createWithCB and/or mamaPublisher_setCallbacks. That way we report the error on the specific publisher. What are your thoughts? Regards, Sam On 14-11-07 10:57 AM, Glenn McClements wrote: Hi Tom, Sam, Before you responded I was considering the use of an inbox as a solution to Sam’s query but I wasn’t overly keen on it. The reason being that being rejected by the broker felt very much like a middleware level event, not an application or market data event so it feels better to me that the error comes from the middleware as a callback. Also, responding with a MamaMsg to the sender means assuming the connection is up and alive, whereas a middleware ACL policy might actually prevent this from happening. It could be that you form the message on the client side in the bridge, but this itself feels incorrect as it’s translating from a middleware level even into a data message. (Also a minor point is that the MAMA_MSG_TYPE_SEC_STATUS relates to security status events on a exchange). This is not to say that a message response is incorrect for RMDS posts however, it’s just a different type of event. For Sam’s point I do prefer the idea of currently returning a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and the transport is currently the best/only place to do this. Going forward we could add callbacks to the MamaPublisher object to inform the client asynchronously of events like this, and MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED seems reasonable though we would need to consider how this would be used/interact with the higher level concept of market data entitlements. Another related concept is guaranteed or acknowledged messaging, which would generate other asynchronous event on a publishers. Like the original issue, this could be done at a high level with MamaMsgs being passed back to acknowledge delivery, but generally this is implemented at the middleware level so again a publisher callback would be better (with a handle to the original message as well for context). Glenn On 07/11/2014 14:20, "Tom Doust" <tom.doust@...> wrote: Hi In the Tick42 RMDS bridge we address this problem by allowing applications to use the sendFromInbox methods on a publisher and where applicable converting the an asynchronous response from the RMDS into a mama message which is delivered to the inbox. Although this is intended to support the RMDS message "post" publishing model that will ACK (or NAK) every message I think it will work for any middleware that has an asynchronous response mechanism; other publishing models on RMDS do not have a message reponse. The message sent to the inbox is arbitrary but it would make sense if other bridges that used this technique used the same message structure. The message we use is formed as follows mamaMsg_addI32(msg, "MdMsgType", 1, MAMA_MSG_TYPE_SEC_STATUS); mamaMsgStatus status = MAMA_MSG_STATUS_OK; if(nakCode != RSSL_NAKC_NONE) { status = RsslNakCode2MamaMsgStatus(nakCode); } mamaMsg_addI32(msg, "MdMsgStatus", 2, status); mamaMsg_addString(msg, "wSymbol",470, symbol_.c_str()); As you can see we just convert an rssl NAK code to an equivalent mama code, or use an OK status in the case of an ACK. These include states like MAMA_MSG_STATUS_NOT_ENTITLED and MAMA_MSG_STATUS_BAD_SYMBOL amongst a number of others. Obviously the set of code and the mappings are platform specific The one thing I think is missing here is a field containing a text string which could carry more information relevant to the client application. We would propose that everyone addressing the issue of processing asynchronous publishing responses takes this approach and that as a community we agree on a convention for the message content. Best Regards Tom Doust -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Sam Wilson Sent: 06 November 2014 7:45 PM To: openmama-dev@... Subject: [Openmama-dev] Asynchronous publisher events Hey all, We're looking for a way to notify a mama application asynchronously of errors while publishing, specifically when the published message was rejected by the message broker's ACL. Our API assumes the messages were accepted, and returns immediately. After some time you might get a connection-level event callback informing the application that its messages were rejected. I have two questions. First off, what is the best way to represent this situation in 2.3.1? We've been tossing around the idea of just tearing down the connection, giving the application a MAMA_TRANSPORT_DISCONNECT or a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and logging the error. Since the application or the environment is incorrectly configured, there's not much that can be done by the application anyways. Secondly, if openmama were to be extended in the future to support these kind of events, what would such extensions look like? Perhaps adding a MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED to the mamaTransportTopicEvent enumeration? Thanks, Sam _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev
More All Messages By This Member
Re: Asynchronous publisher events Sam Wilson <Sam.Wilson@...> #1386 Hey Glenn, Tom, I have to agree with Glenn here. I don't think these kind of events belong alongside regular messages, for a couple of reasons. For one, simply enough, they aren't messages. They don't originate from a publisher, they wouldn't travel along the wire, and they probably would have to be synthesized in the middleware bridge. Secondly, Tom's solution only works for sendFromInbox. The callback mechanism is much more general, and would work for non-inbox messages as well, which is important for our use case. Assuming we do go ahead with implementing callbacks for asynchronous publisher events, what needs to be thought about/discussed before moving on to writing/accepting a patch? I have two ideas on how to implement this, but keep in mind that I'm rather new to OpenMAMA and market data in general. We could use a mamaTransportTopic event, and report the errors with a callback on the transport. The other option that I see, which Glenn hinted at, is that we create a structure similar to mamaMsgCallbacks, called mamaPublisherCallbacks; and a pair of new methods mamaPublisher_createWithCB and/or mamaPublisher_setCallbacks. That way we report the error on the specific publisher. What are your thoughts? Regards, Sam toggle quoted message Show quoted text On 14-11-07 10:57 AM, Glenn McClements wrote: Hi Tom, Sam, Before you responded I was considering the use of an inbox as a solution to Sam’s query but I wasn’t overly keen on it. The reason being that being rejected by the broker felt very much like a middleware level event, not an application or market data event so it feels better to me that the error comes from the middleware as a callback. Also, responding with a MamaMsg to the sender means assuming the connection is up and alive, whereas a middleware ACL policy might actually prevent this from happening. It could be that you form the message on the client side in the bridge, but this itself feels incorrect as it’s translating from a middleware level even into a data message. (Also a minor point is that the MAMA_MSG_TYPE_SEC_STATUS relates to security status events on a exchange). This is not to say that a message response is incorrect for RMDS posts however, it’s just a different type of event. For Sam’s point I do prefer the idea of currently returning a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and the transport is currently the best/only place to do this. Going forward we could add callbacks to the MamaPublisher object to inform the client asynchronously of events like this, and MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED seems reasonable though we would need to consider how this would be used/interact with the higher level concept of market data entitlements. Another related concept is guaranteed or acknowledged messaging, which would generate other asynchronous event on a publishers. Like the original issue, this could be done at a high level with MamaMsgs being passed back to acknowledge delivery, but generally this is implemented at the middleware level so again a publisher callback would be better (with a handle to the original message as well for context). Glenn On 07/11/2014 14:20, "Tom Doust" <tom.doust@...> wrote: Hi In the Tick42 RMDS bridge we address this problem by allowing applications to use the sendFromInbox methods on a publisher and where applicable converting the an asynchronous response from the RMDS into a mama message which is delivered to the inbox. Although this is intended to support the RMDS message "post" publishing model that will ACK (or NAK) every message I think it will work for any middleware that has an asynchronous response mechanism; other publishing models on RMDS do not have a message reponse. The message sent to the inbox is arbitrary but it would make sense if other bridges that used this technique used the same message structure. The message we use is formed as follows mamaMsg_addI32(msg, "MdMsgType", 1, MAMA_MSG_TYPE_SEC_STATUS); mamaMsgStatus status = MAMA_MSG_STATUS_OK; if(nakCode != RSSL_NAKC_NONE) { status = RsslNakCode2MamaMsgStatus(nakCode); } mamaMsg_addI32(msg, "MdMsgStatus", 2, status); mamaMsg_addString(msg, "wSymbol",470, symbol_.c_str()); As you can see we just convert an rssl NAK code to an equivalent mama code, or use an OK status in the case of an ACK. These include states like MAMA_MSG_STATUS_NOT_ENTITLED and MAMA_MSG_STATUS_BAD_SYMBOL amongst a number of others. Obviously the set of code and the mappings are platform specific The one thing I think is missing here is a field containing a text string which could carry more information relevant to the client application. We would propose that everyone addressing the issue of processing asynchronous publishing responses takes this approach and that as a community we agree on a convention for the message content. Best Regards Tom Doust -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Sam Wilson Sent: 06 November 2014 7:45 PM To: openmama-dev@... Subject: [Openmama-dev] Asynchronous publisher events Hey all, We're looking for a way to notify a mama application asynchronously of errors while publishing, specifically when the published message was rejected by the message broker's ACL. Our API assumes the messages were accepted, and returns immediately. After some time you might get a connection-level event callback informing the application that its messages were rejected. I have two questions. First off, what is the best way to represent this situation in 2.3.1? We've been tossing around the idea of just tearing down the connection, giving the application a MAMA_TRANSPORT_DISCONNECT or a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and logging the error. Since the application or the environment is incorrectly configured, there's not much that can be done by the application anyways. Secondly, if openmama were to be extended in the future to support these kind of events, what would such extensions look like? Perhaps adding a MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED to the mamaTransportTopicEvent enumeration? Thanks, Sam _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev
More
Re: Asynchronous publisher events Tom Doust #1385 Hi Glenn In many cases publishing a message is transactional and can succeed or fail. Failure does not mean that the connection (for the transport or the item is at fault) in which case connection level call-backs are not the appropriate way to signal a transaction failure. Even with the ACL failure condition described by Sam could recover without a reconnect from the client if the Solace server supports dynamic changes to the ACL. On RMDS/TREP a common cause for receiving a NAK is that the user is not permissioned to publish this particular symbol on this source. The may be entitled on other symbols I'm inclined to agree with you about the use of callbacks over messages but in practical terms the "send-from-inbox / reply-to-inbox" mechanism is in place and is consistently available in the c/cpp, java and .net apis; that’s not true of call-backs. It's also true that messages are generally easier to manage in an asynchronous, multi-threaded environment. Best regards Tom toggle quoted message Show quoted text -----Original Message----- From: Glenn McClements [mailto:g.mcclements@...] Sent: 07 November 2014 3:57 PM To: Tom Doust; Sam Wilson; openmama-dev@... Subject: Re: [Openmama-dev] Asynchronous publisher events Hi Tom, Sam, Before you responded I was considering the use of an inbox as a solution to Sam’s query but I wasn’t overly keen on it. The reason being that being rejected by the broker felt very much like a middleware level event, not an application or market data event so it feels better to me that the error comes from the middleware as a callback. Also, responding with a MamaMsg to the sender means assuming the connection is up and alive, whereas a middleware ACL policy might actually prevent this from happening. It could be that you form the message on the client side in the bridge, but this itself feels incorrect as it’s translating from a middleware level even into a data message. (Also a minor point is that the MAMA_MSG_TYPE_SEC_STATUS relates to security status events on a exchange). This is not to say that a message response is incorrect for RMDS posts however, it’s just a different type of event. For Sam’s point I do prefer the idea of currently returning a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and the transport is currently the best/only place to do this. Going forward we could add callbacks to the MamaPublisher object to inform the client asynchronously of events like this, and MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED seems reasonable though we would need to consider how this would be used/interact with the higher level concept of market data entitlements. Another related concept is guaranteed or acknowledged messaging, which would generate other asynchronous event on a publishers. Like the original issue, this could be done at a high level with MamaMsgs being passed back to acknowledge delivery, but generally this is implemented at the middleware level so again a publisher callback would be better (with a handle to the original message as well for context). Glenn On 07/11/2014 14:20, "Tom Doust" <tom.doust@...> wrote: Hi In the Tick42 RMDS bridge we address this problem by allowing applications to use the sendFromInbox methods on a publisher and where applicable converting the an asynchronous response from the RMDS into a mama message which is delivered to the inbox. Although this is intended to support the RMDS message "post" publishing model that will ACK (or NAK) every message I think it will work for any middleware that has an asynchronous response mechanism; other publishing models on RMDS do not have a message reponse. The message sent to the inbox is arbitrary but it would make sense if other bridges that used this technique used the same message structure. The message we use is formed as follows mamaMsg_addI32(msg, "MdMsgType", 1, MAMA_MSG_TYPE_SEC_STATUS); mamaMsgStatus status = MAMA_MSG_STATUS_OK; if(nakCode != RSSL_NAKC_NONE) { status = RsslNakCode2MamaMsgStatus(nakCode); } mamaMsg_addI32(msg, "MdMsgStatus", 2, status); mamaMsg_addString(msg, "wSymbol",470, symbol_.c_str()); As you can see we just convert an rssl NAK code to an equivalent mama code, or use an OK status in the case of an ACK. These include states like MAMA_MSG_STATUS_NOT_ENTITLED and MAMA_MSG_STATUS_BAD_SYMBOL amongst a number of others. Obviously the set of code and the mappings are platform specific The one thing I think is missing here is a field containing a text string which could carry more information relevant to the client application. We would propose that everyone addressing the issue of processing asynchronous publishing responses takes this approach and that as a community we agree on a convention for the message content. Best Regards Tom Doust -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Sam Wilson Sent: 06 November 2014 7:45 PM To: openmama-dev@... Subject: [Openmama-dev] Asynchronous publisher events Hey all, We're looking for a way to notify a mama application asynchronously of errors while publishing, specifically when the published message was rejected by the message broker's ACL. Our API assumes the messages were accepted, and returns immediately. After some time you might get a connection-level event callback informing the application that its messages were rejected. I have two questions. First off, what is the best way to represent this situation in 2.3.1? We've been tossing around the idea of just tearing down the connection, giving the application a MAMA_TRANSPORT_DISCONNECT or a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and logging the error. Since the application or the environment is incorrectly configured, there's not much that can be done by the application anyways. Secondly, if openmama were to be extended in the future to support these kind of events, what would such extensions look like? Perhaps adding a MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED to the mamaTransportTopicEvent enumeration? Thanks, Sam _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev
More All Messages By This Member
Re: mamaMsg_toString & mamaMsg_freeString Damian Maguire <d.maguire@...> #1384 Hey Reed, This was actually discussed a while ago on the mailing lists, after a question from Guy. At the time Frank responded and indicated that the mamaMsg_freeString function was deprecated and that the expectation (and current Qpid Proton bridge implementation) is that bridges will reuse a string buffer for performance reasons. You can see Frank's response to the original question in the chain here: http://lists.openmama.org/pipermail/openmama-dev/2013-November/000740.html We also raised a Bugzilla ticket to mark freeString as completely deprecated. This is actually a bit more complex than it sounds, but is set to be resolved as part of the enum removal/vtable work, so we'd expect to get it cleaned up in the reasonably near future. You can actually see how this is to be implemented, and some of the complexity associated with it here: https://github.com/OpenMAMA/OpenMAMA-dynamic/commit/77ae57177c42dbf00f1ba9cf5f858230bb555b83 Let me know if you have any other questions on this. Thanks, Damian On 10/11/14 16:23, Alpert, Reed wrote: toggle quoted message Show quoted text Hi, Question on mamaMsg_toString & mamaMsg_freeString. toString() calls into bridge impl, but freeString() just returns (is not available in the bridge impl mamaPayloadBridgeImpl). Currently for Tick42 malloc is called in the toString(), and the user is expected to call free() when done with the buffer. Is this the design intended for all bridges? Thanks, Reed. Reed Alpert \| Corporate & Investment Bank \| Market Data Services \| J.P. Morgan \| 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 \| T: 718.242.5198 \| M: 917.414.4613 \| reed.alpert@... Alternate Contact: CIB PIM Trading Technology Solutions NA \| CIB_PIM_Trading_Technology_Solutions_NA@... This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.
More All Messages By This Member
mamaMsg_toString & mamaMsg_freeString Alpert, Reed <reed.alpert@...> #1383 Hi, Question on mamaMsg_toString & mamaMsg_freeString. toString() calls into bridge impl, but freeString() just returns (is not available in the bridge impl mamaPayloadBridgeImpl). Currently for Tick42 malloc is called in the toString(), and the user is expected to call free() when done with the buffer. Is this the design intended for all bridges? Thanks, Reed. Reed Alpert \| Corporate & Investment Bank \| Market Data Services \| J.P. Morgan \| 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 \| T: 718.242.5198 \| M: 917.414.4613 \| reed.alpert@... Alternate Contact: CIB PIM Trading Technology Solutions NA \| CIB_PIM_Trading_Technology_Solutions_NA@... This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.
More All Messages By This Member
Re: Asynchronous publisher events Glenn McClements <g.mcclements@...> #1382 Hi Tom, Sam, Before you responded I was considering the use of an inbox as a solution to Sam’s query but I wasn’t overly keen on it. The reason being that being rejected by the broker felt very much like a middleware level event, not an application or market data event so it feels better to me that the error comes from the middleware as a callback. Also, responding with a MamaMsg to the sender means assuming the connection is up and alive, whereas a middleware ACL policy might actually prevent this from happening. It could be that you form the message on the client side in the bridge, but this itself feels incorrect as it’s translating from a middleware level even into a data message. (Also a minor point is that the MAMA_MSG_TYPE_SEC_STATUS relates to security status events on a exchange). This is not to say that a message response is incorrect for RMDS posts however, it’s just a different type of event. For Sam’s point I do prefer the idea of currently returning a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and the transport is currently the best/only place to do this. Going forward we could add callbacks to the MamaPublisher object to inform the client asynchronously of events like this, and MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED seems reasonable though we would need to consider how this would be used/interact with the higher level concept of market data entitlements. Another related concept is guaranteed or acknowledged messaging, which would generate other asynchronous event on a publishers. Like the original issue, this could be done at a high level with MamaMsgs being passed back to acknowledge delivery, but generally this is implemented at the middleware level so again a publisher callback would be better (with a handle to the original message as well for context). Glenn toggle quoted message Show quoted text On 07/11/2014 14:20, "Tom Doust" <tom.doust@...> wrote: Hi In the Tick42 RMDS bridge we address this problem by allowing applications to use the sendFromInbox methods on a publisher and where applicable converting the an asynchronous response from the RMDS into a mama message which is delivered to the inbox. Although this is intended to support the RMDS message "post" publishing model that will ACK (or NAK) every message I think it will work for any middleware that has an asynchronous response mechanism; other publishing models on RMDS do not have a message reponse. The message sent to the inbox is arbitrary but it would make sense if other bridges that used this technique used the same message structure. The message we use is formed as follows mamaMsg_addI32(msg, "MdMsgType", 1, MAMA_MSG_TYPE_SEC_STATUS); mamaMsgStatus status = MAMA_MSG_STATUS_OK; if(nakCode != RSSL_NAKC_NONE) { status = RsslNakCode2MamaMsgStatus(nakCode); } mamaMsg_addI32(msg, "MdMsgStatus", 2, status); mamaMsg_addString(msg, "wSymbol",470, symbol_.c_str()); As you can see we just convert an rssl NAK code to an equivalent mama code, or use an OK status in the case of an ACK. These include states like MAMA_MSG_STATUS_NOT_ENTITLED and MAMA_MSG_STATUS_BAD_SYMBOL amongst a number of others. Obviously the set of code and the mappings are platform specific The one thing I think is missing here is a field containing a text string which could carry more information relevant to the client application. We would propose that everyone addressing the issue of processing asynchronous publishing responses takes this approach and that as a community we agree on a convention for the message content. Best Regards Tom Doust -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Sam Wilson Sent: 06 November 2014 7:45 PM To: openmama-dev@... Subject: [Openmama-dev] Asynchronous publisher events Hey all, We're looking for a way to notify a mama application asynchronously of errors while publishing, specifically when the published message was rejected by the message broker's ACL. Our API assumes the messages were accepted, and returns immediately. After some time you might get a connection-level event callback informing the application that its messages were rejected. I have two questions. First off, what is the best way to represent this situation in 2.3.1? We've been tossing around the idea of just tearing down the connection, giving the application a MAMA_TRANSPORT_DISCONNECT or a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and logging the error. Since the application or the environment is incorrectly configured, there's not much that can be done by the application anyways. Secondly, if openmama were to be extended in the future to support these kind of events, what would such extensions look like? Perhaps adding a MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED to the mamaTransportTopicEvent enumeration? Thanks, Sam _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev
More
Re: Asynchronous publisher events Tom Doust #1381 Hi In the Tick42 RMDS bridge we address this problem by allowing applications to use the sendFromInbox methods on a publisher and where applicable converting the an asynchronous response from the RMDS into a mama message which is delivered to the inbox. Although this is intended to support the RMDS message "post" publishing model that will ACK (or NAK) every message I think it will work for any middleware that has an asynchronous response mechanism; other publishing models on RMDS do not have a message reponse. The message sent to the inbox is arbitrary but it would make sense if other bridges that used this technique used the same message structure. The message we use is formed as follows mamaMsg_addI32(msg, "MdMsgType", 1, MAMA_MSG_TYPE_SEC_STATUS); mamaMsgStatus status = MAMA_MSG_STATUS_OK; if(nakCode != RSSL_NAKC_NONE) { status = RsslNakCode2MamaMsgStatus(nakCode); } mamaMsg_addI32(msg, "MdMsgStatus", 2, status); mamaMsg_addString(msg, "wSymbol",470, symbol_.c_str()); As you can see we just convert an rssl NAK code to an equivalent mama code, or use an OK status in the case of an ACK. These include states like MAMA_MSG_STATUS_NOT_ENTITLED and MAMA_MSG_STATUS_BAD_SYMBOL amongst a number of others. Obviously the set of code and the mappings are platform specific The one thing I think is missing here is a field containing a text string which could carry more information relevant to the client application. We would propose that everyone addressing the issue of processing asynchronous publishing responses takes this approach and that as a community we agree on a convention for the message content. Best Regards Tom Doust toggle quoted message Show quoted text -----Original Message----- From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Sam Wilson Sent: 06 November 2014 7:45 PM To: openmama-dev@... Subject: [Openmama-dev] Asynchronous publisher events Hey all, We're looking for a way to notify a mama application asynchronously of errors while publishing, specifically when the published message was rejected by the message broker's ACL. Our API assumes the messages were accepted, and returns immediately. After some time you might get a connection-level event callback informing the application that its messages were rejected. I have two questions. First off, what is the best way to represent this situation in 2.3.1? We've been tossing around the idea of just tearing down the connection, giving the application a MAMA_TRANSPORT_DISCONNECT or a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and logging the error. Since the application or the environment is incorrectly configured, there's not much that can be done by the application anyways. Secondly, if openmama were to be extended in the future to support these kind of events, what would such extensions look like? Perhaps adding a MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED to the mamaTransportTopicEvent enumeration? Thanks, Sam _______________________________________________ Openmama-dev mailing list Openmama-dev@... https://lists.openmama.org/mailman/listinfo/openmama-dev
More All Messages By This Member
Asynchronous publisher events Sam Wilson <Sam.Wilson@...> #1380 Hey all, We're looking for a way to notify a mama application asynchronously of errors while publishing, specifically when the published message was rejected by the message broker's ACL. Our API assumes the messages were accepted, and returns immediately. After some time you might get a connection-level event callback informing the application that its messages were rejected. I have two questions. First off, what is the best way to represent this situation in 2.3.1? We've been tossing around the idea of just tearing down the connection, giving the application a MAMA_TRANSPORT_DISCONNECT or a MAMA_TRANSPORT_PUBLISHER_DISCONNECT, and logging the error. Since the application or the environment is incorrectly configured, there's not much that can be done by the application anyways. Secondly, if openmama were to be extended in the future to support these kind of events, what would such extensions look like? Perhaps adding a MAMA_TRANSPORT_TOPIC_PUBLISH_DENIED to the mamaTransportTopicEvent enumeration? Thanks, Sam
More

921 - 940 of 2319

Home Hashtags Subgroups Terms