Date   

Discussion Point: Dynamic Bridge Loading and Default payloads

Frank Quinn <f.quinn@...>
 

Hi Folks,

As brought up on a recent thread from Ben:

    2)My only observation, which is somewhat related, is that we kept the
    concept of a default payload. I think this break the whole purpose of
    OpenMama as that means mw bridge must come with their default
    payload, which is loaded by default, irrespective to the fact you are
    going to use it or not. I think this is a defect, and was hoping that we
    will take the occasion of dynamic bridge loading to get this changed.
    Although it might be a big change as it involves probably deprecating
    MamaMsg_create in favour of _createForPayload....

    I'm very keen to get feedback from the community, and particularly
    our user community about this.


I have written both enterprise applications based on OpenMAMA as well as internal OpenMAMA code, and as a user, I think mamaMsg_create() without any payload provided is very handy. I think there are a couple of reasons for this:

  1. mamaMsg_create is one of those functions which you can call from anywhere in your application. It doesn't depend on which bridge you happen to be in during a callback for or anything like that so it can literally come from anywhere. Part of its traditional appeal is specifically that the application developer doesn't need to consider which payloads are in play, and if he / she does care, there is now a separate _createForPayload method which will provide this level of control.
  2. mamaMsg_create usually does exactly what the user will want in almost every scenario. If there is only one middleware involved as it's a one sided API (i.e. a MAMA client or a MAMA publisher), the first payload recommended by the bridge is typically what the user wants to load. It's the same if there is a mid-tier (MAMA pub + sub in one MAMA instance) application where the middlewares are the same north and south.
  3. Deprecating mamaMsg_create will (I expect) have a reasonably large impact on users as a whole. In fact mamaMsg_createforPayload is actually a new method for users moving from MAMA 5 to OpenMAMA, so migrating customers will never have seen it before.
OK so that's why I think mamaMsg_create() is a nice thing to have wearing my "user hat". Here is where I think it's broken and potential solutions wearing my "bridge developer helmet":

  1. It makes the default payload library to be the first payload it finds among all payloads. This is OK in most scenarios, but I think there is a notable exception where there is a multi-bridge app present and the preferred payload for bridge 1 is not even supported by bridge 2. That means any payloads created in this application instance cannot be published by bridge 2 without additional transcoding.
    Potential Solution: Wait until all middleware bridges are loaded before selecting a default payload based on which payload bridges exist across all bridges.
  2. It has no way to be aware of optimal payload selections for each middleware bridge. An example would, again, be in a multi-bridge environment where the algorithm from #1 (let's say) is used to decide a payload compromise, but in the context where a mamaMsg_create is being called, if you always know that it will only ever be create for use with one middleware bridge, you will want to select whichever payload is preferred for that bridge.
    Potential Solution: Have an application-facing way to provide the user with the preferred payload for a given bridge object and then the user can call _createforPayload based on that value.
  3. For middleware bridges that want to support all possible payload bridges that support serialization and deserialization and possibly don't even have their own payload implementation at all (e.g. ZMQ), they may not wish to select a preference for which payload bridge to use at all.
    Potential Solution: Provide a mama.properties parameter which may be used to find supplementary, user-provided compatible bridges.

So I agree with you in that mamaMsg_create is a little broken, but I think they're all fixable without deprecating a pretty handy convenience function for most users.

Cheers,
Frank

The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC


Re: Dynamic Bridge Loading in 2.4.0 and Bridge Backwards Compatibility

Frank Quinn <f.quinn@...>
 

Thanks Ben - glad to see you're happy to take the plunge!

On point #2, I was actually talking to Damian about this very point yesterday and I think I agree with the sentiment of what you're saying, but it's a _little_ off topic as that is related to a customer-facing API change which is a little hairier because it may involve recompiles and possibly code changes on the client side... in saying that, I think there are ways we can alleviate your concerns without needing to change the customer facing interface, so I'll start up a new thread to discuss that.

Cheers,
Frank

On 04/02/16 08:25, Benjamin Taieb wrote:

Hi Frank,

I won't make it to today's call, but wanted to give some feedback :

 

1)I think it is perfectly OK to break backwards compatibility given:

·         We are all aware that the API need some architectural improvement, that will yield further benefit down the line. In this case, we would break compatibility to allow introducing change in the future without breaking backward compatibility (bonus point here if you understood what I meant).

·         Those non backwards compatibles changes are limited in scope (here just bridges), and here only mw vendors/open source bridge are concerned. The user community doesn't change their application code.

 

2)My only observation, which is somewhat related, is that we kept the concept of a default payload. I think this break the whole purpose of OpenMama as that means mw bridge must come with their default payload, which is loaded by default, irrespective to the fact you are going to use it or not. I think this is a defect, and was hoping that we will take the occasion of dynamic bridge loading to get this changed. Although it might be a big change as it involves probably deprecating MamaMsg_create in favour of _createForPayload....

I'm very keen to get feedback from the community, and particularly our user community about this.

 

Cheers,

Ben.

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Frank Quinn
Sent: 03 February 2016 18:18
To: openmama-dev@...
Subject: [Openmama-dev] Dynamic Bridge Loading in 2.4.0 and Bridge Backwards Compatibility

 

Hi Folks,

 

Dynamic Bridge Loading is coming in 2.4.0 and it brings with it a level of flexibility which is new to OpenMAMA’s internal bridges. It effectively allows:

 

1.       OpenMAMA to have the concept of mandatory bridge functions (thereby failing to load incompatible bridges where these bridge functions are not present).

2.       Creation of new and optional bridge functions will no longer break the bridge interface itself.

3.       Reordering of methods in each bridge will no longer break the bridge.

4.       Third parties may create OpenMAMA bridges without having to reserve enum values with the OpenMAMA project.

 

Dynamic bridge loading itself was designed with backwards compatibility in mind, but the more we spoke with users in the community, the more we understood that backwards compatibility in the bridge (as opposed to within the OpenMAMA API itself) is nowhere near as important as avoiding run time crashes due to incompatible bridges being loaded. Note that the upcoming Publisher Events framework in its current form actually depends on this functionality as it involves updating of bridge functions.

 

With that in mind, we would like to propose that the internal bridge interface (again, not to be confused with the OpenMAMA API which will remain unchanged) for OpenMAMA 2.4.x is not backwards compatible with 2.3.x.

 

The impact of this would be that each bridge vendor will need to make a few small changes to work with OpenMAMA 2.4.x as documented over at https://github.com/OpenMAMA/OpenMAMA/wiki/Dynamic-Bridge-Loading. Failure to conform with the new changes will result in OpenMAMA gracefully failing to load the bridge in question at runtime.

 

Taking this stance now will:

 

1.       Reduce the amount of code which third party bridge developers need to maintain.

2.       Ensure that we do not get hit with compatibility issues due to third party bridges not populating the bridge in a way which OpenMAMA expects in future.

3.       Allow us to retire a bunch of code rather than have a large conditional revolving around “if this is not a recent bridge”.

4.       Make it much easier for bridge developers to remain compatible with future OpenMAMA bridge changes going forward.

 

We will be discussing on a technical committee call tomorrow whether or not it’s OK to break backwards compatibility. Depending on the outcome of that, we may be sending around a note to this list offering an opportunity for other members of the community to object before pulling the trigger on this.

 

Cheers,

Frank


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC



The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC


Re: Dynamic Bridge Loading in 2.4.0 and Bridge Backwards Compatibility

Benjamin Taieb
 

Hi Frank,

I won't make it to today's call, but wanted to give some feedback :

 

1)I think it is perfectly OK to break backwards compatibility given:

·         We are all aware that the API need some architectural improvement, that will yield further benefit down the line. In this case, we would break compatibility to allow introducing change in the future without breaking backward compatibility (bonus point here if you understood what I meant).

·         Those non backwards compatibles changes are limited in scope (here just bridges), and here only mw vendors/open source bridge are concerned. The user community doesn't change their application code.

 

2)My only observation, which is somewhat related, is that we kept the concept of a default payload. I think this break the whole purpose of OpenMama as that means mw bridge must come with their default payload, which is loaded by default, irrespective to the fact you are going to use it or not. I think this is a defect, and was hoping that we will take the occasion of dynamic bridge loading to get this changed. Although it might be a big change as it involves probably deprecating MamaMsg_create in favour of _createForPayload....

I'm very keen to get feedback from the community, and particularly our user community about this.

 

Cheers,

Ben.

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Frank Quinn
Sent: 03 February 2016 18:18
To: openmama-dev@...
Subject: [Openmama-dev] Dynamic Bridge Loading in 2.4.0 and Bridge Backwards Compatibility

 

Hi Folks,

 

Dynamic Bridge Loading is coming in 2.4.0 and it brings with it a level of flexibility which is new to OpenMAMA’s internal bridges. It effectively allows:

 

1.       OpenMAMA to have the concept of mandatory bridge functions (thereby failing to load incompatible bridges where these bridge functions are not present).

2.       Creation of new and optional bridge functions will no longer break the bridge interface itself.

3.       Reordering of methods in each bridge will no longer break the bridge.

4.       Third parties may create OpenMAMA bridges without having to reserve enum values with the OpenMAMA project.

 

Dynamic bridge loading itself was designed with backwards compatibility in mind, but the more we spoke with users in the community, the more we understood that backwards compatibility in the bridge (as opposed to within the OpenMAMA API itself) is nowhere near as important as avoiding run time crashes due to incompatible bridges being loaded. Note that the upcoming Publisher Events framework in its current form actually depends on this functionality as it involves updating of bridge functions.

 

With that in mind, we would like to propose that the internal bridge interface (again, not to be confused with the OpenMAMA API which will remain unchanged) for OpenMAMA 2.4.x is not backwards compatible with 2.3.x.

 

The impact of this would be that each bridge vendor will need to make a few small changes to work with OpenMAMA 2.4.x as documented over at https://github.com/OpenMAMA/OpenMAMA/wiki/Dynamic-Bridge-Loading. Failure to conform with the new changes will result in OpenMAMA gracefully failing to load the bridge in question at runtime.

 

Taking this stance now will:

 

1.       Reduce the amount of code which third party bridge developers need to maintain.

2.       Ensure that we do not get hit with compatibility issues due to third party bridges not populating the bridge in a way which OpenMAMA expects in future.

3.       Allow us to retire a bunch of code rather than have a large conditional revolving around “if this is not a recent bridge”.

4.       Make it much easier for bridge developers to remain compatible with future OpenMAMA bridge changes going forward.

 

We will be discussing on a technical committee call tomorrow whether or not it’s OK to break backwards compatibility. Depending on the outcome of that, we may be sending around a note to this list offering an opportunity for other members of the community to object before pulling the trigger on this.

 

Cheers,

Frank


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC


Dynamic Bridge Loading in 2.4.0 and Bridge Backwards Compatibility

Frank Quinn <f.quinn@...>
 

Hi Folks,

 

Dynamic Bridge Loading is coming in 2.4.0 and it brings with it a level of flexibility which is new to OpenMAMA’s internal bridges. It effectively allows:

 

1.       OpenMAMA to have the concept of mandatory bridge functions (thereby failing to load incompatible bridges where these bridge functions are not present).

2.       Creation of new and optional bridge functions will no longer break the bridge interface itself.

3.       Reordering of methods in each bridge will no longer break the bridge.

4.       Third parties may create OpenMAMA bridges without having to reserve enum values with the OpenMAMA project.

 

Dynamic bridge loading itself was designed with backwards compatibility in mind, but the more we spoke with users in the community, the more we understood that backwards compatibility in the bridge (as opposed to within the OpenMAMA API itself) is nowhere near as important as avoiding run time crashes due to incompatible bridges being loaded. Note that the upcoming Publisher Events framework in its current form actually depends on this functionality as it involves updating of bridge functions.

 

With that in mind, we would like to propose that the internal bridge interface (again, not to be confused with the OpenMAMA API which will remain unchanged) for OpenMAMA 2.4.x is not backwards compatible with 2.3.x.

 

The impact of this would be that each bridge vendor will need to make a few small changes to work with OpenMAMA 2.4.x as documented over at https://github.com/OpenMAMA/OpenMAMA/wiki/Dynamic-Bridge-Loading. Failure to conform with the new changes will result in OpenMAMA gracefully failing to load the bridge in question at runtime.

 

Taking this stance now will:

 

1.       Reduce the amount of code which third party bridge developers need to maintain.

2.       Ensure that we do not get hit with compatibility issues due to third party bridges not populating the bridge in a way which OpenMAMA expects in future.

3.       Allow us to retire a bunch of code rather than have a large conditional revolving around “if this is not a recent bridge”.

4.       Make it much easier for bridge developers to remain compatible with future OpenMAMA bridge changes going forward.

 

We will be discussing on a technical committee call tomorrow whether or not it’s OK to break backwards compatibility. Depending on the outcome of that, we may be sending around a note to this list offering an opportunity for other members of the community to object before pulling the trigger on this.

 

Cheers,

Frank


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC


Re: Openmama Documentation for subscription onDestroy

cmorgan
 

I’ve raised an issue to cover this on github.

 

Here is a link to the issue:

https://github.com/OpenMAMA/OpenMAMA/issues/114

 

Chris Morgan

 

 

From: Alpert, Reed [mailto:reed.alpert@...]
Sent: February-01-16 10:48 AM
To: Duane Pauls <Duane.Pauls@...>; Christopher Morgan <Christopher.Morgan@...>; openmama-dev@...
Subject: RE: Openmama Documentation for subscription onDestroy

 

I think the app needs to call sub->destroy(), then wait for the onDestroy(), and then ‘delete sub’.

 

onDestroy() is a completion event – once you get it you won’t hear from that subscription again (no callbacks of any kind).

 

In order for an app to call C++ or Java ‘delete sub’ w/o sub->destroy the destructor needs to completely disconnect the app from the impl (which as Duane & Chris point out it does not).

In that case the inline onDestroy() does have value in that the app knows it can safely destroy any of its data structures.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Duane Pauls
Sent: Thursday, January 28, 2016 1:39 PM
To: Christopher Morgan; openmama-dev@...
Subject: Re: [Openmama-dev] Openmama Documentation for subscription onDestroy

 

I’ll add a bit of clarification to the problem with the CPP interface if the middleware bridge queues an event to call the onDestroy callback.

 

When an application destructs a MamaSubscription object, the application onDestroy is called inline, and the MamaSubscription object is freed, but the corresponding MamaSubscriptionImpl object is not freed.  So far so good – it is the MamaSubscriptionImpl object pointer that is the closure to the C-layer and this is still valid, unfreed memory.  However, at this point the MamaSubscriptionImpl object’s mCallback pointer still points to the application’s MamaSubscriptionCallback object.  The application may have freed this memory since MamaSubscriptionCallback::onDestroy() has been called.

 

If we look at MamdaSubscription as an example, it in fact does not implement MamaSubscriptionCallback::onDestroy at all.  It will always destroy the callback (MamdaSubscription::mImpl) immediately after destroying the underlying MamaSubscription.  This happens to occur after onDestroy is called since it is called inline with MamaSubscription::~MamaSubscription.  But nevertheless, it is freed immediately after onDestroy is called inline with the call to MamaSubscription::~MamaSubscription().

 

Now, sometime later, the C-layer onDestroy is called.  This calls the static MamaSubscription::onSubscriptionDestroy which calls MamaSubscriptionImpl::InvokeDestroy().  This checks if MamaSubscriptionImpl::mCallback is NULL (it is not – in the case of a MamdaSubscription it is still pointing to a freed MamdaSubscriptionImpl object) and tries to invoke the onDestroy() method.  If the vtable pointer is no longer valid, a core here is quite likely.

 

Even if a core does not occur, the application would see onDestroy() twice – once inline with when the destroy was started and a second time when the lower layer destroy was completed.  It seems the design intent here is to decouple the MamaSubscriptionImpl from everything above it when the MamaSubscription is destroyed.  To do this, it seems the right thing to do in the C++ layer would be to set MamaSubscriptionImpl::mCallback to NULL when setting MamaSubscriptionImpl::mFreed to true (or alternatively include mFreed in the tests that check mCallback == NULL).

 

Should a bug be raised for this issue?  Or should a middleware bridge in fact NOT enqueue the onDestroy callback as stated in the OpenMAMA Bridge Documentation Wiki and instead invoke onDestroy inline?  If the intent is to always invoke the callback inline, it seems the benefit of having a callback in the first place is diminished.

 

Cheers,

Duane

 

From: Christopher Morgan
Sent: Thursday, January 28, 2016 10:30 AM
To:
openmama-dev@...
Subject: Openmama Documentation for subscription onDestroy

 

Hi,

 

I’m with solace dev for the openmama middle bridge and I have question about mamaSubscription onDestroy with the CPP openmama. On the openmama Developer Wiki under Bridge Documentation for myMiddlewareBridgeMamaSubscription_destroy () it states “…the destroy method adds an event to the queue to invokes the destroy callback, which notifies the client application of the successful destroy of the subscription.” Our Bridge currently does this but in the CPP API for openmama the MamaSubscription seems to do an inline onDestroy as a part of the destructor. This does not work well with the enqueuerd onDestroy callback (potential for causing segmentation faults). I’ve also compared to the Qpid bridge which seems to do an inline onDestroy callback as a part their MiddlewareBridgeMamaSubscription_destroy(). Should the onDestroy callback be inline the MiddlewareBridgeMamaSubscription_destroy() like the Qpid to work with the CPP API? Or Is there a problem with the CPP API?

 

Chris Morgan

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Re: Openmama Documentation for subscription onDestroy

Alpert, Reed <reed.alpert@...>
 

I think the app needs to call sub->destroy(), then wait for the onDestroy(), and then ‘delete sub’.

 

onDestroy() is a completion event – once you get it you won’t hear from that subscription again (no callbacks of any kind).

 

In order for an app to call C++ or Java ‘delete sub’ w/o sub->destroy the destructor needs to completely disconnect the app from the impl (which as Duane & Chris point out it does not).

In that case the inline onDestroy() does have value in that the app knows it can safely destroy any of its data structures.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Duane Pauls
Sent: Thursday, January 28, 2016 1:39 PM
To: Christopher Morgan; openmama-dev@...
Subject: Re: [Openmama-dev] Openmama Documentation for subscription onDestroy

 

I’ll add a bit of clarification to the problem with the CPP interface if the middleware bridge queues an event to call the onDestroy callback.

 

When an application destructs a MamaSubscription object, the application onDestroy is called inline, and the MamaSubscription object is freed, but the corresponding MamaSubscriptionImpl object is not freed.  So far so good – it is the MamaSubscriptionImpl object pointer that is the closure to the C-layer and this is still valid, unfreed memory.  However, at this point the MamaSubscriptionImpl object’s mCallback pointer still points to the application’s MamaSubscriptionCallback object.  The application may have freed this memory since MamaSubscriptionCallback::onDestroy() has been called.

 

If we look at MamdaSubscription as an example, it in fact does not implement MamaSubscriptionCallback::onDestroy at all.  It will always destroy the callback (MamdaSubscription::mImpl) immediately after destroying the underlying MamaSubscription.  This happens to occur after onDestroy is called since it is called inline with MamaSubscription::~MamaSubscription.  But nevertheless, it is freed immediately after onDestroy is called inline with the call to MamaSubscription::~MamaSubscription().

 

Now, sometime later, the C-layer onDestroy is called.  This calls the static MamaSubscription::onSubscriptionDestroy which calls MamaSubscriptionImpl::InvokeDestroy().  This checks if MamaSubscriptionImpl::mCallback is NULL (it is not – in the case of a MamdaSubscription it is still pointing to a freed MamdaSubscriptionImpl object) and tries to invoke the onDestroy() method.  If the vtable pointer is no longer valid, a core here is quite likely.

 

Even if a core does not occur, the application would see onDestroy() twice – once inline with when the destroy was started and a second time when the lower layer destroy was completed.  It seems the design intent here is to decouple the MamaSubscriptionImpl from everything above it when the MamaSubscription is destroyed.  To do this, it seems the right thing to do in the C++ layer would be to set MamaSubscriptionImpl::mCallback to NULL when setting MamaSubscriptionImpl::mFreed to true (or alternatively include mFreed in the tests that check mCallback == NULL).

 

Should a bug be raised for this issue?  Or should a middleware bridge in fact NOT enqueue the onDestroy callback as stated in the OpenMAMA Bridge Documentation Wiki and instead invoke onDestroy inline?  If the intent is to always invoke the callback inline, it seems the benefit of having a callback in the first place is diminished.

 

Cheers,

Duane

 

From: Christopher Morgan
Sent: Thursday, January 28, 2016 10:30 AM
To:
openmama-dev@...
Subject: Openmama Documentation for subscription onDestroy

 

Hi,

 

I’m with solace dev for the openmama middle bridge and I have question about mamaSubscription onDestroy with the CPP openmama. On the openmama Developer Wiki under Bridge Documentation for myMiddlewareBridgeMamaSubscription_destroy () it states “…the destroy method adds an event to the queue to invokes the destroy callback, which notifies the client application of the successful destroy of the subscription.” Our Bridge currently does this but in the CPP API for openmama the MamaSubscription seems to do an inline onDestroy as a part of the destructor. This does not work well with the enqueuerd onDestroy callback (potential for causing segmentation faults). I’ve also compared to the Qpid bridge which seems to do an inline onDestroy callback as a part their MiddlewareBridgeMamaSubscription_destroy(). Should the onDestroy callback be inline the MiddlewareBridgeMamaSubscription_destroy() like the Qpid to work with the CPP API? Or Is there a problem with the CPP API?

 

Chris Morgan

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Re: Openmama Documentation for subscription onDestroy

Duane Pauls <Duane.Pauls@...>
 

I’ll add a bit of clarification to the problem with the CPP interface if the middleware bridge queues an event to call the onDestroy callback.

 

When an application destructs a MamaSubscription object, the application onDestroy is called inline, and the MamaSubscription object is freed, but the corresponding MamaSubscriptionImpl object is not freed.  So far so good – it is the MamaSubscriptionImpl object pointer that is the closure to the C-layer and this is still valid, unfreed memory.  However, at this point the MamaSubscriptionImpl object’s mCallback pointer still points to the application’s MamaSubscriptionCallback object.  The application may have freed this memory since MamaSubscriptionCallback::onDestroy() has been called.

 

If we look at MamdaSubscription as an example, it in fact does not implement MamaSubscriptionCallback::onDestroy at all.  It will always destroy the callback (MamdaSubscription::mImpl) immediately after destroying the underlying MamaSubscription.  This happens to occur after onDestroy is called since it is called inline with MamaSubscription::~MamaSubscription.  But nevertheless, it is freed immediately after onDestroy is called inline with the call to MamaSubscription::~MamaSubscription().

 

Now, sometime later, the C-layer onDestroy is called.  This calls the static MamaSubscription::onSubscriptionDestroy which calls MamaSubscriptionImpl::InvokeDestroy().  This checks if MamaSubscriptionImpl::mCallback is NULL (it is not – in the case of a MamdaSubscription it is still pointing to a freed MamdaSubscriptionImpl object) and tries to invoke the onDestroy() method.  If the vtable pointer is no longer valid, a core here is quite likely.

 

Even if a core does not occur, the application would see onDestroy() twice – once inline with when the destroy was started and a second time when the lower layer destroy was completed.  It seems the design intent here is to decouple the MamaSubscriptionImpl from everything above it when the MamaSubscription is destroyed.  To do this, it seems the right thing to do in the C++ layer would be to set MamaSubscriptionImpl::mCallback to NULL when setting MamaSubscriptionImpl::mFreed to true (or alternatively include mFreed in the tests that check mCallback == NULL).

 

Should a bug be raised for this issue?  Or should a middleware bridge in fact NOT enqueue the onDestroy callback as stated in the OpenMAMA Bridge Documentation Wiki and instead invoke onDestroy inline?  If the intent is to always invoke the callback inline, it seems the benefit of having a callback in the first place is diminished.

 

Cheers,

Duane

 

From: Christopher Morgan
Sent: Thursday, January 28, 2016 10:30 AM
To: openmama-dev@...
Subject: Openmama Documentation for subscription onDestroy

 

Hi,

 

I’m with solace dev for the openmama middle bridge and I have question about mamaSubscription onDestroy with the CPP openmama. On the openmama Developer Wiki under Bridge Documentation for myMiddlewareBridgeMamaSubscription_destroy () it states “…the destroy method adds an event to the queue to invokes the destroy callback, which notifies the client application of the successful destroy of the subscription.” Our Bridge currently does this but in the CPP API for openmama the MamaSubscription seems to do an inline onDestroy as a part of the destructor. This does not work well with the enqueuerd onDestroy callback (potential for causing segmentation faults). I’ve also compared to the Qpid bridge which seems to do an inline onDestroy callback as a part their MiddlewareBridgeMamaSubscription_destroy(). Should the onDestroy callback be inline the MiddlewareBridgeMamaSubscription_destroy() like the Qpid to work with the CPP API? Or Is there a problem with the CPP API?

 

Chris Morgan


Openmama Documentation for subscription onDestroy

cmorgan
 

Hi,

 

I’m with solace dev for the openmama middle bridge and I have question about mamaSubscription onDestroy with the CPP openmama. On the openmama Developer Wiki under Bridge Documentation for myMiddlewareBridgeMamaSubscription_destroy () it states “…the destroy method adds an event to the queue to invokes the destroy callback, which notifies the client application of the successful destroy of the subscription.” Our Bridge currently does this but in the CPP API for openmama the MamaSubscription seems to do an inline onDestroy as a part of the destructor. This does not work well with the enqueuerd onDestroy callback (potential for causing segmentation faults). I’ve also compared to the Qpid bridge which seems to do an inline onDestroy callback as a part their MiddlewareBridgeMamaSubscription_destroy(). Should the onDestroy callback be inline the MiddlewareBridgeMamaSubscription_destroy() like the Qpid to work with the CPP API? Or Is there a problem with the CPP API?

 

Chris Morgan


FW: [Openmama-ci] Code change(s) just landed on origin/next (Fixed)

Frank Quinn <f.quinn@...>
 

Hi Folks,

 

As discussed in the technical committee meeting, an example of the notifications I was considering subscribing openmama-dev to is listed below:

 

They should only get triggered when a change lands on either ‘next’ or ‘master’ branches.


Currently these notifications are only sent to members of the openmama-ci mailing list, but if we can collapse the two without adding too much noise, then the openmama-ci mailing list can be removed.

 

Cheers,

Frank

 

From: openmama-ci-bounces@... [mailto:openmama-ci-bounces@...] On Behalf Of jenkins@...
Sent: 12 January 2016 15:15
To: openmama-ci@...
Subject: [Openmama-ci] Code change(s) just landed on origin/next (Fixed)

 

Some changes have just been added to the origin/next branch!

 
[Matt Mulhern] UNITTEST: disabling invalid msg test.
        mama/c_cpp/src/gunittest/c/mamamsg/msggeneraltests.cpp
 
 

Results for OpenMAMA Next Branch (Qpid Proton) CI run with latest changes:

 

  • CI Project Name: OpenMAMA Next Branch (Qpid Proton)
  • Build Number: #8
  • Build Status: Fixed
  • Total Amount of Tests: 1762
  • Passed: 1762
  • Failed: 0
  • Skipped / Disabled: 0

You may also check CI console output to view the full results.


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC


Re: Feature freeze for OpenMAMA 2.4.0

Frank Quinn <f.quinn@...>
 

Hi Folks,

 

After a few extra days grace, I’m now declaring the feature set for OpenMAMA 2.4.0 frozen. Work will be completed on the outstanding issues to be merged into next, then the 2.4.0 code cut will be taken.

 

Also note that Dynamic Bridge Loading has just landed into the next branch (and Phil just fixed OSX to work with it too which we’re about to merge in).

 

Cheers,

Frank

 

From: Frank Quinn
Sent: 06 January 2016 13:11
To: 'openmama-dev@...' <openmama-dev@...>
Subject: Feature freeze for OpenMAMA 2.4.0

 

Hi Folks,

 

Happy new year!

 

Now, let's round up the features / bug fixes that are going to make it into the next OpenMAMA release (2.4.0).

 

Are there any existing issues that are not listed below that anyone would like to ensure is included in the next release? If so, please reply to this email on-list with the github ticket number and we'll see if it's feasible to squeeze them in.

 

Features in progress for being included in next release:

 

·         Publisher Events (Under Development)

·         Dynamic Entitlements for OEA abstraction (Under Development)

·         Dynamic Bridge Loading  (Code Review)

·         MSVC 2015 Support (Code Review)

 

Summary of existing changes already completed for next release (full list available at https://github.com/OpenMAMA/OpenMAMA/compare/master...next):

 

·         Qpid proton broker support

·         Several qpid proton bridge bug fixes

·         CentOS / RHEL 7 support

·         Timer fixes

·         New Unit tests

·         Several changes to work with recent OSX

·         Avis fixes to provide a clean CI run (See the clouds over at http://ci.openmama.org)

 

I will aggregate and appraise responses until Wednesday 13th January, at which point I will send another email advising when the 2.4.0 cut will be taken.

 

Cheers,

Frank


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC


Feature freeze for OpenMAMA 2.4.0

Frank Quinn <f.quinn@...>
 

Hi Folks,

 

Happy new year!

 

Now, let's round up the features / bug fixes that are going to make it into the next OpenMAMA release (2.4.0).

 

Are there any existing issues that are not listed below that anyone would like to ensure is included in the next release? If so, please reply to this email on-list with the github ticket number and we'll see if it's feasible to squeeze them in.

 

Features in progress for being included in next release:

 

·         Publisher Events (Under Development)

·         Dynamic Entitlements for OEA abstraction (Under Development)

·         Dynamic Bridge Loading  (Code Review)

·         MSVC 2015 Support (Code Review)

 

Summary of existing changes already completed for next release (full list available at https://github.com/OpenMAMA/OpenMAMA/compare/master...next):

 

·         Qpid proton broker support

·         Several qpid proton bridge bug fixes

·         CentOS / RHEL 7 support

·         Timer fixes

·         New Unit tests

·         Several changes to work with recent OSX

·         Avis fixes to provide a clean CI run (See the clouds over at http://ci.openmama.org)

 

I will aggregate and appraise responses until Wednesday 13th January, at which point I will send another email advising when the 2.4.0 cut will be taken.

 

Cheers,

Frank


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC


MamaSubscription java finalizer will never run

Alpert, Reed <reed.alpert@...>
 

Hi,

 

MAMAJNI: MamaSubscription has unnecessary finalizer which impacts memory #94

 

MamaSubscription java class has a finalizer which won't be called until after destroy() is called since destroy deletes the global ref in jni back to the java object. But if destroy is called then calling finalizer does nothing as it detects that destroy has already been called. The downside is that for a sub churn app each MamaSub is put on the finalizer queue, which delays the GC of the object (finalizer thread typically runs with low priority).

 

Example is the pub/sub churn app (MamaPublisher does not have finalizer) I am running. The MamaPubs on the heap vary from count of 500 - 15k, depending on how recently GC has run. The MamaSubs vary count between 10k and 40k, never getting below 10k, even just after GC has run. A heap dump downs the MamaSubs being held by the finalizer thread.

 

I propose that the finalizer in MamaSub be removed, since it will never run, and impacts the free space.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Question on publisher events and transport topic callbacks

Alpert, Reed <reed.alpert@...>
 

Hi,

 

I’ve been working through the publisher events via transport topics callbacks with Solace dev.

We have a few questions and are looking for input from the dev community.

 

Briefly, the publisher events support 2 ways of returning events:

1.       Via callbacks passed in by the app and returning the mamaPublisher

2.       Via transport topic callbacks in cases where the mamaPublisher is not available

 

Bridges can use mamaTransportImpl_invokeTransportTopicCallback() to call the mama layer to make these callbacks.

However there is no queue available for these to allow the app to determine which thread the callbacks will be returned in.

 

I can add a new method to set the topic callbacks that includes a queue, and make the queue available to the bridges.

extern mama_status

mamaTransport_setTransportTopicCallback2 (mamaTransport transport,

                                          mamaTransportTopicCB callback,

                                          mamaQueue queue,

                                          void* closure);

extern mama_status

mamaTransport_getTransportTopicQueue (mamaTransport transport, mamaQueue* queue);

 

Or the queue can be added in the transport create (new create for back compat) to be available for both transport cb and topic cb and other uses.

 

Please let me know any preferences.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Re: OpenMAMA Publisher Events RFC

Frank Quinn <f.quinn@...>
 

Hi Folks,

 

Happy to report that no amendments were reported as required within the required timeframe, so we may now consider this RFC closed.

 

Thanks Reed – looking forward to the pull request.

 

Cheers,

Frank

 

From: Frank Quinn
Sent: 02 November 2015 08:56
To: 'Glenn McClements' <g.mcclements@...>; Alpert, Reed <reed.alpert@...>; openmama-dev@...
Subject: RE: [Openmama-dev] OpenMAMA Publisher Events RFC

 

Hi Folks,

 

Note that as per the notes from the workshops in September, the normal RFC window for response is typically 2 weeks, which we are well past at this point.

 

In saying that, as this is the first RFC since that decision was made, we’ll leave this open until the end of the week (6th November), at which point we will consider this RFC approved.

 

Cheers,

Frank

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Glenn McClements
Sent: 28 October 2015 17:51
To: Alpert, Reed <reed.alpert@...>; openmama-dev@...
Subject: Re: [Openmama-dev] OpenMAMA Publisher Events RFC

 

Folks, please spend some time reviewing this if you get a chance. We’ve gone though quite a few iterations of this so this is the final sign off.

 

Cheers,

Glenn 

 

Glenn McClements, SVP Engineering 

Adelaide Exchange, 24-26 Adelaide Street,  Belfast, UK, BT2 8GD

www.srtechlabs.com

 

SR.LABS Proven High Speed Electronic Trading Solutions

 

From: <openmama-dev-bounces@...> on behalf of "Alpert, Reed via Openmama-dev" <openmama-dev@...>
Reply-To: "Alpert, Reed" <reed.alpert@...>
Date: Friday, 9 October 2015 16:26
To: "openmama-dev@..." <openmama-dev@...>
Cc: CIB Market Data Services Engineers <CIB_Market_Data_Services_Engineers@...>
Subject: [Openmama-dev] OpenMAMA Publisher Events RFC

 

Hi,

 

Attached is an RFC for OpenMAMA Publisher Events.

 

Please send any comments/suggestions/etc.

 

Publisher events provides a way for bridges to provide events back to the application.

An example is permission_denied for a publish, but not limited to that.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC


Re: OpenMAMA Publisher Events RFC

Frank Quinn <f.quinn@...>
 

Hi Folks,

 

Note that as per the notes from the workshops in September, the normal RFC window for response is typically 2 weeks, which we are well past at this point.

 

In saying that, as this is the first RFC since that decision was made, we’ll leave this open until the end of the week (6th November), at which point we will consider this RFC approved.

 

Cheers,

Frank

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Glenn McClements
Sent: 28 October 2015 17:51
To: Alpert, Reed <reed.alpert@...>; openmama-dev@...
Subject: Re: [Openmama-dev] OpenMAMA Publisher Events RFC

 

Folks, please spend some time reviewing this if you get a chance. We’ve gone though quite a few iterations of this so this is the final sign off.

 

Cheers,

Glenn 

 

Glenn McClements, SVP Engineering 

Adelaide Exchange, 24-26 Adelaide Street,  Belfast, UK, BT2 8GD

www.srtechlabs.com

 

SR.LABS Proven High Speed Electronic Trading Solutions

 

From: <openmama-dev-bounces@...> on behalf of "Alpert, Reed via Openmama-dev" <openmama-dev@...>
Reply-To: "Alpert, Reed" <reed.alpert@...>
Date: Friday, 9 October 2015 16:26
To: "openmama-dev@..." <openmama-dev@...>
Cc: CIB Market Data Services Engineers <CIB_Market_Data_Services_Engineers@...>
Subject: [Openmama-dev] OpenMAMA Publisher Events RFC

 

Hi,

 

Attached is an RFC for OpenMAMA Publisher Events.

 

Please send any comments/suggestions/etc.

 

Publisher events provides a way for bridges to provide events back to the application.

An example is permission_denied for a publish, but not limited to that.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC


Re: OpenMAMA Publisher Events RFC

Glenn McClements <g.mcclements@...>
 

Folks, please spend some time reviewing this if you get a chance. We’ve gone though quite a few iterations of this so this is the final sign off.

Cheers,
Glenn 


Glenn McClements, SVP Engineering 

Adelaide Exchange, 24-26 Adelaide Street,  Belfast, UK, BT2 8GD

www.srtechlabs.com

 

SR.LABS Proven High Speed Electronic Trading Solutions


From: <openmama-dev-bounces@...> on behalf of "Alpert, Reed via Openmama-dev" <openmama-dev@...>
Reply-To: "Alpert, Reed" <reed.alpert@...>
Date: Friday, 9 October 2015 16:26
To: "openmama-dev@..." <openmama-dev@...>
Cc: CIB Market Data Services Engineers <CIB_Market_Data_Services_Engineers@...>
Subject: [Openmama-dev] OpenMAMA Publisher Events RFC

Hi,

 

Attached is an RFC for OpenMAMA Publisher Events.

 

Please send any comments/suggestions/etc.

 

Publisher events provides a way for bridges to provide events back to the application.

An example is permission_denied for a publish, but not limited to that.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC


Re: [Openmama-users] Problem bringing fields from bloomberg professional terminal

Tom Doust
 

Hi Nestor

 

Do you have mappings for these fields in the field map file?

 

Feel free to mail me a copy of the file directly and I will take a look

 

 

Best regards

 

Tom

 

 

TOM DOUST | Head of Consultancy                                                                                                         


TICK42

P: +44 (0) 1628 477444 | M: +44 (0) 7710 479924 | E: tom.doust@... | skype:  tom.doust |  http://www.tick42.com

 

 

From: openmama-users-bounces@... [mailto:openmama-users-bounces@...] On Behalf Of Macrux
Sent: 26 October 2015 15:26
To: openmama-dev@...; openmama-users@...
Subject: [Openmama-users] Problem bringing fields from bloomberg professional terminal

 

Hi there,

I have a problem bringing some fields with the tick42blp bridge, I'm trying to get the data for the price levels (wBestBidPrice1, wBestBidPrice2,..., wBestBidPrice10) as well as the sizes for those levels and also for the ask side, but when the mamalisten example receives the MamaMsg, it doesn't has those fields, even if they are specified as the documentation shows:

mamalistenc -m tick42blp -tport blp_tport -S BlpMktData -s "AAPL US Equity,[BEST_BID1, BEST_BID2,BEST_BID3,BEST_ASK1,BEST_ASK2,BEST_ASK3]"

And so on...

I have the fieldmap.csv file in WOMBAT_PATH, and I'm using Blomberg's professional terminal, in fact, I can bring these fields using the example included in Blomberg Open API, but not through the bridge which brings other fields like wBidPrice, wAskPrice, wBidSize, wAskSize, wEventTime, among others, but not information about levels.

Has anyone had this  issue?

Thanks in advance,

Nestor.


Problem bringing fields from bloomberg professional terminal

macrux
 

Hi there,

I have a problem bringing some fields with the tick42blp bridge, I'm trying to get the data for the price levels (wBestBidPrice1, wBestBidPrice2,..., wBestBidPrice10) as well as the sizes for those levels and also for the ask side, but when the mamalisten example receives the MamaMsg, it doesn't has those fields, even if they are specified as the documentation shows:

mamalistenc -m tick42blp -tport blp_tport -S BlpMktData -s "AAPL US Equity,[BEST_BID1, BEST_BID2,BEST_BID3,BEST_ASK1,BEST_ASK2,BEST_ASK3]"

And so on...

I have the fieldmap.csv file in WOMBAT_PATH, and I'm using Blomberg's professional terminal, in fact, I can bring these fields using the example included in Blomberg Open API, but not through the bridge which brings other fields like wBidPrice, wAskPrice, wBidSize, wAskSize, wEventTime, among others, but not information about levels.

Has anyone had this  issue?

Thanks in advance,

Nestor.


Notes from Workshop Session 24-Sep-2015

Frank Quinn <f.quinn@...>
 

Hi Folks,

 

First of all a special thanks to everyone who attended attend our last workshop session on 24th September which was focused around the complementary topics of Source Discovery, Publisher Events and Local Data Sharing. Once again I am happy to report it was a very productive session and we covered a lot of ground!

 

Glenn and I have aggregated some notes from the session which are listed below - any feedback is welcome.

 

Source Discovery

 

Implementation Notes:

 

·         Client applications should be notified when new sources become available. There is also a requirement for client applications to be notified of state changes of sources. At the very minimum “UP” and “DOWN” states but existing OpenMAMA status should be reused where appropriate. We also need to include in this a clear definition of each state e.g. what “STALE” actually means.

 

·         Other metadata for sources is a requirement. This data should be flexible and consist of a well-known set of key-value pairs. Existing OpenMAMA mechanisms such as messages, field caches or properties could be reused to cache the meta data. Also, the concept of an initial message, with the full set of metadata fields, and then delta messages after that, should be used and closely mirror how regular subscriptions work in OpenMAMA. 

 

Proposed standard metadata for sources includes:

 

-          Name

-          State

-          Capability list

-          Transport

-          Data Dictionary

-          Symbology

 

·         Sources should be able to be queried on a per bridge and transport basis. Specifically, the topology shall be defined as:

 

Bridge -> Transport -> Source -> Group -> Publisher -> Topic

 

Note that the concept of Group isn’t currently defined within OpenMAMA, but should provide a useful and logical way of grouping symbols together.

 

·         An OpenMAMA API implementing the above should be defined, with options of the implementation being an “OpenMAMA native” source discovery solution, or by another market data platform. A flexible solution to this would use OpenMAMA messages as a way of passing source information back and forth between the client application and the middleware bridge implementations.

 

·         The existing MamaSourceManager object can be extended to store the metadata, provide callbacks and implement the rest of the functionality above. Note that MamaSourceManager will store all sources available to an application and not be tied to one particular bridge. This would be optional to use, and would intercept messages and cache data in the messages from the source discovery implementation.

 

·         For an OpenMAMA native implementation of source discovery, it should be optional to have a central lookup point on the platform. Also, if implemented using standard OpenMAMA pub/sub semantics, then multi-tier solutions, i.e. Ones with some sort of OpenMAMA proxy, should also work.

 

·         The basic mechanisms should be based on OpenMAMA messages, with a listener in place to cache the messages and their fields for later query via a new OpenMAMA source manager method. This listener can also propagate the event through to the calling application if it has registered interest in receiving such updates.

 

·         Transport callbacks may be used to advise the MAMA Source Manager when a source or group of sources have gone offline

 

 

Related Notes:

 

·         Source discovery is particularly useful for screen based application.

 

·         Topic discovery is out of scope for this implementation.

 

·         There is an existing SR Labs commercial component called Worldview which provides a limited ability to query sources. Because of dependencies it is not possible to open source this at the moment, however the underlying protocol to serialise and deserialise sources, which is all based on OpenMAMA, could be open sourced and used as a basis for the above.

 

 

Local Data Sharing

 

Implementation Notes:

 

·         One of the major differences between this and existing forms of OpenMAMA publishing, is the concept of fan in, or multiple active publishers on the same topic.  This currently will not work with OpenMAMA because sequence numbers must be unique and contiguous per topic (per transport). Therefore, the major requirement/change is that there is an option, on a per source basis, that sequence numbers should be considered on a per topic + publisher basis.  The mechanism to provide this variation in data quality validation could leverage the OpenMAMA plugin framework discussed in the previous workshop.

 

·         Another difference from existing publishing is the optional requirement for acknowledged delivery of messages. There is the possibility that the ACK would be delivered via a bridge or using OpenMAMA native constructs such as an inbox, but the callbacks implemented for Publisher Events should be used to notify the user.

 

·         Currently when OpenMAMA makes a non-basic subscription it will time out after a configurable number of initial image requests and delays. For local data sharing there will need to be an option to wait indefinitely. 

 

·         A gateway or proxy server to aggregate published messages should not be a requirement, however it may be used within certain deployments to provide a layer enabling auditing, caching, authentication etc.

 

·         Again, this should be defined and implemented in a top down manner, with an OpenMAMA API implementing the above features, with pluggable implementations.

 

Related Notes:

 

·         Suggestions made to call contributions a “Message / Notice Board” rather than “contributions”

 

·         Other implementations send such contributed back up through the subscription. As an implementation like this would be excessively middleware dependant, OpenMAMA should not adopt this approach.

 

·         Roundtrip subscription to contributed data is a simple and foolproof way to ensure that the contributed data has been propagated.

 

 

Publisher Events

 

Implementation Notes

 

·         General agreement of the design but a desire to get it distributed to the OpenMAMA community in the for of RFC, with a period of 2 weeks for feedback.

 

·         A bridge interface change is required, so this would be dependent on the new bridge loading mechanism discussed in the first workshop.

 

·         Guaranteed messaging agreed as out of scope

 

 

Tangents and Wish Lists

 

·         Question asked around lag to be expected for getting OpenMAMA changes into Enterprise MAMA 6. Confirmed it should take around 1-2 months from the cut taken depending on successful testing.

 

·         There was a query to see if anything could be done to make joining the Linux Foundation easier for large financial institutions.

 

·         The same mechanism used for source metadata detection could be used to propagate market events such as trading statuses which are currently sent on a per topic basis.

 

·         A mechanism closely related to source discovery is topic discovery on a per source basis. OpenMAMA currently has a feature called symbol lists which satisfies some of this requirement but there are limitations with it including lack of multiple announce topics. Ideally a new mechanism should be provided which allows a sub set of symbols from a particular source to be queried i.e. symbol groups, with symbols being added, updated or deleted dynamically. There may also be a requirement for this list to be ordered in some way. The logic for this is similar to how order books are handled in OpenMAMDA.

 

 

Cheers,

Frank & Glenn

 


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC


Re: Inconsistencies/bugs with OpenMama

Sridharan, Venkatesan <Venkatesan.Sridharan@...>
 

Thanks Frank. Have raised a github issue for each point.

-----Original Message-----
From: Frank Quinn [mailto:f.quinn@srtechlabs.com]
Sent: Monday, October 12, 2015 3:22 PM
To: Sridharan, Venkatesan <Venkatesan.Sridharan@Squarepoint-Capital.com>; openmama-dev@lists.openmama.org
Subject: RE: Inconsistencies/bugs with OpenMama

Hi Venkatesan,

Glad you're getting something from the project and appreciate your efforts around this!

At first look, I agree in principle with a lot of the below, but I feel like it could get unruly trying to cover everything in a single thread. With that in mind, could you raise a github issue for each of the below so we can track each point in detail independently and make sure nothing slips through the net?

Also note there was also a series of patches submitted in the past to cover datetime / price vectors which may address at least some of your concerns, though we were holding off on implementing any bridge-related changes until dynamic bridge loading is completed:

http://lists.celinuxforum.org/pipermail/openmama-dev/2014-January/001002.html
http://lists.celinuxforum.org/pipermail/openmama-dev/2014-January/001003.html
http://lists.celinuxforum.org/pipermail/openmama-dev/2014-January/001004.html
http://lists.celinuxforum.org/pipermail/openmama-dev/2014-January/001005.html
http://lists.celinuxforum.org/pipermail/openmama-dev/2014-January/001006.html

Cheers,
Frank

-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Sridharan, Venkatesan
Sent: 12 October 2015 01:59
To: openmama-dev@lists.openmama.org
Subject: [Openmama-dev] Inconsistencies/bugs with OpenMama

Hi Folks,

This is my first post here. First off, thanks for your efforts on OpenMama, the code is clean, well commented and the documentation is great too!

We've been using OpenMama for a few months now and have written bridges for TibRv and Qpid (using qpid c++ API). I am talking to my company about opensourcing them. I came across a number of seeming errors/inconsistencies and wanted to get the community's take. I can provide a patch to fix the issues if we agree that they indeed are bugs and that they do need to be fixed.

* xxxxBridge_getDefaultPayloadId (char ***name, char **id) The function retrieves the payload id which are const and hence params must be const qualified -> (const char ***name, const char **id).

* xxxPayload_getMsg(const msgPayload msg, const char* name, mama_fid_t fid, msgPayload* result) We are passing in a const msg and extracting its field. so, last arg should be const msgPayload*

* loadbridge uses property file before open() During creation of the default queue, properties subsystem is accessed and this always prints an error/warning if we use a custom properties file.

* bridgeMamaPublisherSendReplyToInbox => request is void* instead of mamaMsg typedef mama_status (*bridgeMamaPublisher_sendReplyToInbox)(publisherBridge publisher,
=>void * request, <=
mamaMsg reply);
This is needless as the request is a mamaMsg


* throw in dtor!
MamaTransport::~MamaTransport (void)
{
if (mPimpl)
{
if ( mamaInternal_getCatchCallbackExceptions() && mPimpl->mCallback)
{
delete mPimpl->mCallback;
}
if (mPimpl->mDeleteCTransport)
{
=>mamaTry (mamaTransport_destroy (mTransport)); <=
}
delete mPimpl;
mPimpl = NULL;
}
}

* Bridges are expected to allocate memory for some functions (toString, getReplyHandle etc) but there is no hook to free the memory. Since each bridge is allowed to use custom memory allocation, there should be a hook from mamaMsg_freeString back to the bridge. Also, currently mamaMsg_freeString is a no-op

* mamaMsg_copy does not check for self copy

* Some fields are not initialized in INITIALIZE_BRIDGE () macro. Eg, mamaBridgeImpl::mInternalDispatcher is not set to NULL on bridge creation. Got a crash on this one.

* As per comments and documentation, mamaQueue_timedDispatch is supposed to keep dispatching until the given timeout has elapsed. But, the current bridge implementations use wombatQueue_timedDispatch which dispatches only one message. So, EITHER wombatQueue_timedDispatch should be changed to keep dispatching till timeout OR another helper function should be provided to do this (as this functionality is used across bridges)

* msgPayload_unSerialize prototype incorrect (takes in void** instead of void*) typedef mama_status
(*msgPayload_unSerialize) (const msgPayload msg,
=>const void** buffer, <=
mama_size_t bufferLength);
This is an input buffer and the only function using this is mamaMsg_setNewBuffer which calls the above function and passes a void* buffer as a void**

* Incorrect test in MamaMsg::setNewBuffer ():
bool MamaMsg::setNewBuffer (
void* buffer,
mama_size_t size)
{
int result = 0;
// should be MAMA_STATUS_OK == ...
=> if (MAMA_STATUS_OK != (mamaTry (mamaMsg_setNewBuffer ( <=
mMsg, buffer, size))))
{
return result;
}
return result != 0;
}

* These functions are not defined, I encountered this when trying to build python, perl interfaces with swig. A default implementation returning MAMA_STATUS_NOT_IMPLEMENTED or nullptr should be added mamaMsg_updateVectorTime, mamaMsg_updateVectorPrice, mamaDispatcher_getQueue, mama_forceLogVaWithPrefix, mamaMsgField_getVectorBool, mama_logPolicyToString, mama_tryStringToLogPolicy, mamaMsgIterator_end

* Inconsistent prototypes :
mama_status
mamaMsg_updateVectorPrice (
mamaMsg msg,
const char* fname,
mama_fid_t fid,
//should be const mamaPrice priceList[]
=> const mamaPrice* priceList[], =<
mama_size_t numElements);

msgPayload_getPrice, msgPayload_getDateTime should take pointer to result as last parameter
Eg:
typedef mama_status
(*msgPayload_getDateTime) (const msgPayload msg,
const char* name,
mama_fid_t fid,
//should be mamaDateTime * result
=> mamaDateTime result); <=

* There are some pesky macros defined in port.h (close, sleep etc) which causes problems when using other libraries (eg boost). We should be defining them as inline functions instead of using macros


Cheers,
Venkatesan




Confidentiality Note: This e-mail and any attachments are confidential and may be protected by legal privilege. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this e-mail or any attachment is prohibited. If you have received this e-mail in error, please notify us immediately by returning it to the sender and delete this copy from your system. Thank you for your cooperation.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev

The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. SR Labs LLC



Confidentiality Note: This e-mail and any attachments are confidential and may
be protected by legal privilege. If you are not the intended recipient, be aware
that any disclosure, copying, distribution or use of this e-mail or any
attachment is prohibited. If you have received this e-mail in error, please
notify us immediately by returning it to the sender and delete this copy from
your system. Thank you for your cooperation.

661 - 680 of 2305