Enforcing field type on publish


Alpert, Reed <reed.alpert@...>
 

Hi,

 

This is regarding a way to check that the field data type being published matches (closely) the dictionary data type.

 

The motivation for this is to prevent our internally published data caches from invalid data.

For example, if a field (name=MARK_PRICE fid=300) is a MamaPrice in the dictionary, and an app uses this:

                addString("MARK_PRICE", 300, "invalid price");

then we have a field that is now a string in the cache, and that causes problems with subscribing apps that use getPrice().

 

Currently we are looking to the bridges that we use (Tick42/TREP & Solace) to enforce these restrictions.

 

Adding this to the OpenMAMA layer is tricky, but allows a single policy to be enforced across multiple bridges. The main question from my point of view being how to set the policy. They can range from none (default), to most strict (published field's type must match dictionary), to common conversions supported (e.g., F32 -> F64), to a custom grid setting exactly what conversions are allowed.

 

I welcome any input from others who have run into this same issue, or found other ways to solve it.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Glenn McClements <gmcclements@...>
 

Hi Guy,
When you refer to 'addString' below you mention a cache, but this is not a method of the C/C++MamaFieldCache are you refering to your own cache object or actually a MamaMsg object?

Regards,
Glenn 


From: openmama-dev-bounces@... [openmama-dev-bounces@...] on behalf of Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 13:59
To: Openmama-dev@...
Subject: [Openmama-dev] Enforcing field type on publish

Hi,

 

This is regarding a way to check that the field data type being published matches (closely) the dictionary data type.

 

The motivation for this is to prevent our internally published data caches from invalid data.

For example, if a field (name=MARK_PRICE fid=300) is a MamaPrice in the dictionary, and an app uses this:

                addString("MARK_PRICE", 300, "invalid price");

then we have a field that is now a string in the cache, and that causes problems with subscribing apps that use getPrice().

 

Currently we are looking to the bridges that we use (Tick42/TREP & Solace) to enforce these restrictions.

 

Adding this to the OpenMAMA layer is tricky, but allows a single policy to be enforced across multiple bridges. The main question from my point of view being how to set the policy. They can range from none (default), to most strict (published field's type must match dictionary), to common conversions supported (e.g., F32 -> F64), to a custom grid setting exactly what conversions are allowed.

 

I welcome any input from others who have run into this same issue, or found other ways to solve it.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


Alpert, Reed <reed.alpert@...>
 

Hi,

 

Yes, the addString() is from MamaMsg.

 

The cache is our own market data infrastructure, not the MamaFieldCache.

In this case it is either Solace/SolCache or TREP RTIC.

 

We want business app dev groups to be able to look at the dictionary, see a field that is MamaPrice, and be guaranteed that MamaMsg.getPrice() will always work (although MamaPrice.getIsValidPrice() may return false).

A certain number of the type mismatches come from Excel, where a failed formula may populate a cell as #NA, and that gets published into a price/float field.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Wednesday, September 03, 2014 3:10 PM
To: Alpert, Reed; Openmama-dev@...
Subject: RE: Enforcing field type on publish

 

Hi Guy,

When you refer to 'addString' below you mention a cache, but this is not a method of the C/C++MamaFieldCache

are you refering to your own cache object or actually a MamaMsg object?

 

Regards,

Glenn 

 


From: openmama-dev-bounces@... [openmama-dev-bounces@...] on behalf of Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 13:59
To: Openmama-dev@...
Subject: [Openmama-dev] Enforcing field type on publish

Hi,

 

This is regarding a way to check that the field data type being published matches (closely) the dictionary data type.

 

The motivation for this is to prevent our internally published data caches from invalid data.

For example, if a field (name=MARK_PRICE fid=300) is a MamaPrice in the dictionary, and an app uses this:

                addString("MARK_PRICE", 300, "invalid price");

then we have a field that is now a string in the cache, and that causes problems with subscribing apps that use getPrice().

 

Currently we are looking to the bridges that we use (Tick42/TREP & Solace) to enforce these restrictions.

 

Adding this to the OpenMAMA layer is tricky, but allows a single policy to be enforced across multiple bridges. The main question from my point of view being how to set the policy. They can range from none (default), to most strict (published field's type must match dictionary), to common conversions supported (e.g., F32 -> F64), to a custom grid setting exactly what conversions are allowed.

 

I welcome any input from others who have run into this same issue, or found other ways to solve it.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Glenn McClements <gmcclements@...>
 

There are a number of places where this *could* be done, but not all make sense to me:

- In the payload library - this doesn't feel right as I feel the payload should be 'dumb' and just trust the caller. Also it would need to be done for each supported payload. 

- In the payload bridge - same points as above

- In MamaMsg - this would work across all supported payloads, but again I think the payload should be trust the caller as to what fields should be what type and act a a simple wrapper 

- In a field cache (e.g. the C & C++ MamaFieldCache) - this makes the most sense to me as the logic for maintaining state about fields and if they have changed is already there, so extra rules and checks would belong here

With the current MamaFieldCache you could do this already if you primed the fields by creating them with the correct type on startup, this would depend on having a well known set of fields though because you don't want to be adding everything in the dictionary. You could also try this technique with the MamaMsg object, but it will be payload specific because some will throw an error if you try to update a field with a new type, and others may replace it.

Glenn 



From: Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 15:28
To: Glenn McClements; Openmama-dev@...
Subject: RE: Enforcing field type on publish

Hi,

 

Yes, the addString() is from MamaMsg.

 

The cache is our own market data infrastructure, not the MamaFieldCache.

In this case it is either Solace/SolCache or TREP RTIC.

 

We want business app dev groups to be able to look at the dictionary, see a field that is MamaPrice, and be guaranteed that MamaMsg.getPrice() will always work (although MamaPrice.getIsValidPrice() may return false).

A certain number of the type mismatches come from Excel, where a failed formula may populate a cell as #NA, and that gets published into a price/float field.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Wednesday, September 03, 2014 3:10 PM
To: Alpert, Reed; Openmama-dev@...
Subject: RE: Enforcing field type on publish

 

Hi Guy,

When you refer to 'addString' below you mention a cache, but this is not a method of the C/C++MamaFieldCache

are you refering to your own cache object or actually a MamaMsg object?

 

Regards,

Glenn 

 


From: openmama-dev-bounces@... [openmama-dev-bounces@...] on behalf of Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 13:59
To: Openmama-dev@...
Subject: [Openmama-dev] Enforcing field type on publish

Hi,

 

This is regarding a way to check that the field data type being published matches (closely) the dictionary data type.

 

The motivation for this is to prevent our internally published data caches from invalid data.

For example, if a field (name=MARK_PRICE fid=300) is a MamaPrice in the dictionary, and an app uses this:

                addString("MARK_PRICE", 300, "invalid price");

then we have a field that is now a string in the cache, and that causes problems with subscribing apps that use getPrice().

 

Currently we are looking to the bridges that we use (Tick42/TREP & Solace) to enforce these restrictions.

 

Adding this to the OpenMAMA layer is tricky, but allows a single policy to be enforced across multiple bridges. The main question from my point of view being how to set the policy. They can range from none (default), to most strict (published field's type must match dictionary), to common conversions supported (e.g., F32 -> F64), to a custom grid setting exactly what conversions are allowed.

 

I welcome any input from others who have run into this same issue, or found other ways to solve it.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


Frank Quinn <fquinn.ni@...>
 

Guessing by your capitalization, I'm guessing this is C++? If so, have you considered mandating that all function calls use MamaFieldDescriptors pulled from a standardized dictionary rather than using string / fid pairs?

Cheers,
Frank


On Thu, Sep 4, 2014 at 9:33 PM, Glenn McClements <gmcclements@...> wrote:
There are a number of places where this *could* be done, but not all make sense to me:

- In the payload library - this doesn't feel right as I feel the payload should be 'dumb' and just trust the caller. Also it would need to be done for each supported payload. 

- In the payload bridge - same points as above

- In MamaMsg - this would work across all supported payloads, but again I think the payload should be trust the caller as to what fields should be what type and act a a simple wrapper 

- In a field cache (e.g. the C & C++ MamaFieldCache) - this makes the most sense to me as the logic for maintaining state about fields and if they have changed is already there, so extra rules and checks would belong here

With the current MamaFieldCache you could do this already if you primed the fields by creating them with the correct type on startup, this would depend on having a well known set of fields though because you don't want to be adding everything in the dictionary. You could also try this technique with the MamaMsg object, but it will be payload specific because some will throw an error if you try to update a field with a new type, and others may replace it.

Glenn 



From: Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 15:28
To: Glenn McClements; Openmama-dev@...

Subject: RE: Enforcing field type on publish

Hi,

 

Yes, the addString() is from MamaMsg.

 

The cache is our own market data infrastructure, not the MamaFieldCache.

In this case it is either Solace/SolCache or TREP RTIC.

 

We want business app dev groups to be able to look at the dictionary, see a field that is MamaPrice, and be guaranteed that MamaMsg.getPrice() will always work (although MamaPrice.getIsValidPrice() may return false).

A certain number of the type mismatches come from Excel, where a failed formula may populate a cell as #NA, and that gets published into a price/float field.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Wednesday, September 03, 2014 3:10 PM
To: Alpert, Reed; Openmama-dev@...
Subject: RE: Enforcing field type on publish

 

Hi Guy,

When you refer to 'addString' below you mention a cache, but this is not a method of the C/C++MamaFieldCache

are you refering to your own cache object or actually a MamaMsg object?

 

Regards,

Glenn 

 


From: openmama-dev-bounces@... [openmama-dev-bounces@...] on behalf of Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 13:59
To: Openmama-dev@...
Subject: [Openmama-dev] Enforcing field type on publish

Hi,

 

This is regarding a way to check that the field data type being published matches (closely) the dictionary data type.

 

The motivation for this is to prevent our internally published data caches from invalid data.

For example, if a field (name=MARK_PRICE fid=300) is a MamaPrice in the dictionary, and an app uses this:

                addString("MARK_PRICE", 300, "invalid price");

then we have a field that is now a string in the cache, and that causes problems with subscribing apps that use getPrice().

 

Currently we are looking to the bridges that we use (Tick42/TREP & Solace) to enforce these restrictions.

 

Adding this to the OpenMAMA layer is tricky, but allows a single policy to be enforced across multiple bridges. The main question from my point of view being how to set the policy. They can range from none (default), to most strict (published field's type must match dictionary), to common conversions supported (e.g., F32 -> F64), to a custom grid setting exactly what conversions are allowed.

 

I welcome any input from others who have run into this same issue, or found other ways to solve it.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev



Alpert, Reed <reed.alpert@...>
 

Yes, this is C++, but we also have Java and DotNet clients (some C# and some Excel).

We do publish best practices, but our app dev groups sometimes do not follow them, and we want to prevent the SolCache from getting out of sync with the dictionary.

We also provide them with sample apps that have this type of control, but sometimes … J

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Thursday, September 04, 2014 4:56 PM
To: Glenn McClements
Cc: Alpert, Reed; Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Guessing by your capitalization, I'm guessing this is C++? If so, have you considered mandating that all function calls use MamaFieldDescriptors pulled from a standardized dictionary rather than using string / fid pairs?

Cheers,
Frank

 

On Thu, Sep 4, 2014 at 9:33 PM, Glenn McClements <gmcclements@...> wrote:

There are a number of places where this *could* be done, but not all make sense to me:

 

- In the payload library - this doesn't feel right as I feel the payload should be 'dumb' and just trust the caller. Also it would need to be done for each supported payload. 

 

- In the payload bridge - same points as above

 

- In MamaMsg - this would work across all supported payloads, but again I think the payload should be trust the caller as to what fields should be what type and act a a simple wrapper 

 

- In a field cache (e.g. the C & C++ MamaFieldCache) - this makes the most sense to me as the logic for maintaining state about fields and if they have changed is already there, so extra rules and checks would belong here

 

With the current MamaFieldCache you could do this already if you primed the fields by creating them with the correct type on startup, this would depend on having a well known set of fields though because you don't want to be adding everything in the dictionary. You could also try this technique with the MamaMsg object, but it will be payload specific because some will throw an error if you try to update a field with a new type, and others may replace it.

 

Glenn 

 

 

 

From: Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 15:28
To: Glenn McClements; Openmama-dev@...


Subject: RE: Enforcing field type on publish

 

Hi,

 

Yes, the addString() is from MamaMsg.

 

The cache is our own market data infrastructure, not the MamaFieldCache.

In this case it is either Solace/SolCache or TREP RTIC.

 

We want business app dev groups to be able to look at the dictionary, see a field that is MamaPrice, and be guaranteed that MamaMsg.getPrice() will always work (although MamaPrice.getIsValidPrice() may return false).

A certain number of the type mismatches come from Excel, where a failed formula may populate a cell as #NA, and that gets published into a price/float field.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Wednesday, September 03, 2014 3:10 PM
To: Alpert, Reed; Openmama-dev@...
Subject: RE: Enforcing field type on publish

 

Hi Guy,

When you refer to 'addString' below you mention a cache, but this is not a method of the C/C++MamaFieldCache

are you refering to your own cache object or actually a MamaMsg object?

 

Regards,

Glenn 

 


From: openmama-dev-bounces@... [openmama-dev-bounces@...] on behalf of Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 13:59
To: Openmama-dev@...
Subject: [Openmama-dev] Enforcing field type on publish

Hi,

 

This is regarding a way to check that the field data type being published matches (closely) the dictionary data type.

 

The motivation for this is to prevent our internally published data caches from invalid data.

For example, if a field (name=MARK_PRICE fid=300) is a MamaPrice in the dictionary, and an app uses this:

                addString("MARK_PRICE", 300, "invalid price");

then we have a field that is now a string in the cache, and that causes problems with subscribing apps that use getPrice().

 

Currently we are looking to the bridges that we use (Tick42/TREP & Solace) to enforce these restrictions.

 

Adding this to the OpenMAMA layer is tricky, but allows a single policy to be enforced across multiple bridges. The main question from my point of view being how to set the policy. They can range from none (default), to most strict (published field's type must match dictionary), to common conversions supported (e.g., F32 -> F64), to a custom grid setting exactly what conversions are allowed.

 

I welcome any input from others who have run into this same issue, or found other ways to solve it.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.



_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Benjamin Taieb
 

Just to put emphasis on what Reed explain:

It should be probably implemented in the C layer, making sure that it "catches" all languages "for free".

The goal is to enforce it in OpenMama layer, so that OpenMama application are forced to adheres to it.

 

I believe use of MamaFieldCache is optional, which would rules it out.

And referring to Glenn comments about mamaMsg, the goal is that the payload does not trust the calling application.

 

There is multiples challenges I believe in implementing that:

-How does the mama layer get dictionary ? This could be easily solves by using dictionnary requestor wherever we decides to implement it.

-How does the mapping between a particular msg and namespace is going to work ? At the moment, a dictionary is attached to namespace, namespace could be shared across multiples sources, but a message is source agnostic, only the publisher when it send the msg will send on a particular source/ symbol. On the other side, we would want to fail as soon as possible in the process, probably in the mamaMsg_addXX and mamaMsg_setXX calls, so that the calling application know for which fields it is doing something wrong.

-We need to clarify the "authorised cast": If you're doing addI32 on a I64 fields, should probably get authorised. At the moment, various Payload implement various level of flexibility, would be good that we defines UnitTests defining precisely what it should be, probably modelled from what is defined in wombatmsg, for retro-compatibility purpose.

 

I'll take any input from the community, particularly on point 2, which is the hardest in my opinion.

Cheers,

Ben.

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Alpert, Reed
Sent: 08 September 2014 20:10
To: fquinn.ni@...; Glenn McClements
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Yes, this is C++, but we also have Java and DotNet clients (some C# and some Excel).

We do publish best practices, but our app dev groups sometimes do not follow them, and we want to prevent the SolCache from getting out of sync with the dictionary.

We also provide them with sample apps that have this type of control, but sometimes … J

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Thursday, September 04, 2014 4:56 PM
To: Glenn McClements
Cc: Alpert, Reed; Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Guessing by your capitalization, I'm guessing this is C++? If so, have you considered mandating that all function calls use MamaFieldDescriptors pulled from a standardized dictionary rather than using string / fid pairs?

Cheers,
Frank

 

On Thu, Sep 4, 2014 at 9:33 PM, Glenn McClements <gmcclements@...> wrote:

There are a number of places where this *could* be done, but not all make sense to me:

 

- In the payload library - this doesn't feel right as I feel the payload should be 'dumb' and just trust the caller. Also it would need to be done for each supported payload. 

 

- In the payload bridge - same points as above

 

- In MamaMsg - this would work across all supported payloads, but again I think the payload should be trust the caller as to what fields should be what type and act a a simple wrapper 

 

- In a field cache (e.g. the C & C++ MamaFieldCache) - this makes the most sense to me as the logic for maintaining state about fields and if they have changed is already there, so extra rules and checks would belong here

 

With the current MamaFieldCache you could do this already if you primed the fields by creating them with the correct type on startup, this would depend on having a well known set of fields though because you don't want to be adding everything in the dictionary. You could also try this technique with the MamaMsg object, but it will be payload specific because some will throw an error if you try to update a field with a new type, and others may replace it.

 

Glenn 

 

 

 

From: Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 15:28
To: Glenn McClements; Openmama-dev@...


Subject: RE: Enforcing field type on publish

 

Hi,

 

Yes, the addString() is from MamaMsg.

 

The cache is our own market data infrastructure, not the MamaFieldCache.

In this case it is either Solace/SolCache or TREP RTIC.

 

We want business app dev groups to be able to look at the dictionary, see a field that is MamaPrice, and be guaranteed that MamaMsg.getPrice() will always work (although MamaPrice.getIsValidPrice() may return false).

A certain number of the type mismatches come from Excel, where a failed formula may populate a cell as #NA, and that gets published into a price/float field.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Wednesday, September 03, 2014 3:10 PM
To: Alpert, Reed; Openmama-dev@...
Subject: RE: Enforcing field type on publish

 

Hi Guy,

When you refer to 'addString' below you mention a cache, but this is not a method of the C/C++MamaFieldCache

are you refering to your own cache object or actually a MamaMsg object?

 

Regards,

Glenn 

 


From: openmama-dev-bounces@... [openmama-dev-bounces@...] on behalf of Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 13:59
To: Openmama-dev@...
Subject: [Openmama-dev] Enforcing field type on publish

Hi,

 

This is regarding a way to check that the field data type being published matches (closely) the dictionary data type.

 

The motivation for this is to prevent our internally published data caches from invalid data.

For example, if a field (name=MARK_PRICE fid=300) is a MamaPrice in the dictionary, and an app uses this:

                addString("MARK_PRICE", 300, "invalid price");

then we have a field that is now a string in the cache, and that causes problems with subscribing apps that use getPrice().

 

Currently we are looking to the bridges that we use (Tick42/TREP & Solace) to enforce these restrictions.

 

Adding this to the OpenMAMA layer is tricky, but allows a single policy to be enforced across multiple bridges. The main question from my point of view being how to set the policy. They can range from none (default), to most strict (published field's type must match dictionary), to common conversions supported (e.g., F32 -> F64), to a custom grid setting exactly what conversions are allowed.

 

I welcome any input from others who have run into this same issue, or found other ways to solve it.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.



_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Frank Quinn <fquinn@...>
 

Hi Ben,

 

There are a few things to consider here:

 

1.       The MAMA Dictionary is optional too. If you look at basic mamapublisherc / mamasubscriberc, you’ll see it doesn’t use the dictionary, but it does use mamaMsg.

2.       Unit tests actually do exist for the various mandatory castings required. If you run the payload unit tests today, for example, you’ll find that getting a date time from a string as well as a date time field should be supported, as well as other payload level casting expectations… In fact if you haven’t run these tests already, you really should, especially if you want to use MAMDA. If you look at the Qpid bridge, you’ll see a bunch of code put in to make this casting more forgiving.

3.       While iterating over a message, in the interest of performance, you probably don’t want to hit the data dictionary on every field being processed, particularly if you’re trying to do high performance field caching.

 

Basically, I’m of the opinion that if you want to make a payload flexible with respect to the field types it supports, that should be done at a payload layer and we have unit tests to verify the mandatory minimum.

 

If you want to enforce MAMA Dictionary compliance, this should be done in the MAMA application, where you can use field descriptors to verify content type at send and receive sides. E.g.

 

switch (mamaFieldDescriptor_getType (fieldDescriptor))

{

    case MAMA_FIELD_TYPE_I32:

    case MAMA_FIELD_TYPE_I64:

        allow the add / access;

    default:

        handle condition;

}

 

As well as everything else, whether or not the I64 to I32 casting (yes - the other way around) will be legal will depend on the data content being cast and sent across. Depending on the content, a U8 / U16 could even be fine. Without knowledge and consideration of this within the application, undesired truncation of data could be found, or opportunities to avoid failed data transfer due to strictness checking could be avoided.

 

Cheers,

Frank

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Benjamin Taieb
Sent: 10 September 2014 08:47
To: Alpert, Reed; fquinn.ni@...; Glenn McClements
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Just to put emphasis on what Reed explain:

It should be probably implemented in the C layer, making sure that it "catches" all languages "for free".

The goal is to enforce it in OpenMama layer, so that OpenMama application are forced to adheres to it.

 

I believe use of MamaFieldCache is optional, which would rules it out.

And referring to Glenn comments about mamaMsg, the goal is that the payload does not trust the calling application.

 

There is multiples challenges I believe in implementing that:

-How does the mama layer get dictionary ? This could be easily solves by using dictionnary requestor wherever we decides to implement it.

-How does the mapping between a particular msg and namespace is going to work ? At the moment, a dictionary is attached to namespace, namespace could be shared across multiples sources, but a message is source agnostic, only the publisher when it send the msg will send on a particular source/ symbol. On the other side, we would want to fail as soon as possible in the process, probably in the mamaMsg_addXX and mamaMsg_setXX calls, so that the calling application know for which fields it is doing something wrong.

-We need to clarify the "authorised cast": If you're doing addI32 on a I64 fields, should probably get authorised. At the moment, various Payload implement various level of flexibility, would be good that we defines UnitTests defining precisely what it should be, probably modelled from what is defined in wombatmsg, for retro-compatibility purpose.

 

I'll take any input from the community, particularly on point 2, which is the hardest in my opinion.

Cheers,

Ben.

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Alpert, Reed
Sent: 08 September 2014 20:10
To: fquinn.ni@...; Glenn McClements
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Yes, this is C++, but we also have Java and DotNet clients (some C# and some Excel).

We do publish best practices, but our app dev groups sometimes do not follow them, and we want to prevent the SolCache from getting out of sync with the dictionary.

We also provide them with sample apps that have this type of control, but sometimes … J

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Thursday, September 04, 2014 4:56 PM
To: Glenn McClements
Cc: Alpert, Reed; Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Guessing by your capitalization, I'm guessing this is C++? If so, have you considered mandating that all function calls use MamaFieldDescriptors pulled from a standardized dictionary rather than using string / fid pairs?

Cheers,
Frank

 

On Thu, Sep 4, 2014 at 9:33 PM, Glenn McClements <gmcclements@...> wrote:

There are a number of places where this *could* be done, but not all make sense to me:

 

- In the payload library - this doesn't feel right as I feel the payload should be 'dumb' and just trust the caller. Also it would need to be done for each supported payload. 

 

- In the payload bridge - same points as above

 

- In MamaMsg - this would work across all supported payloads, but again I think the payload should be trust the caller as to what fields should be what type and act a a simple wrapper 

 

- In a field cache (e.g. the C & C++ MamaFieldCache) - this makes the most sense to me as the logic for maintaining state about fields and if they have changed is already there, so extra rules and checks would belong here

 

With the current MamaFieldCache you could do this already if you primed the fields by creating them with the correct type on startup, this would depend on having a well known set of fields though because you don't want to be adding everything in the dictionary. You could also try this technique with the MamaMsg object, but it will be payload specific because some will throw an error if you try to update a field with a new type, and others may replace it.

 

Glenn 

 

 

 

From: Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 15:28
To: Glenn McClements; Openmama-dev@...


Subject: RE: Enforcing field type on publish

 

Hi,

 

Yes, the addString() is from MamaMsg.

 

The cache is our own market data infrastructure, not the MamaFieldCache.

In this case it is either Solace/SolCache or TREP RTIC.

 

We want business app dev groups to be able to look at the dictionary, see a field that is MamaPrice, and be guaranteed that MamaMsg.getPrice() will always work (although MamaPrice.getIsValidPrice() may return false).

A certain number of the type mismatches come from Excel, where a failed formula may populate a cell as #NA, and that gets published into a price/float field.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Wednesday, September 03, 2014 3:10 PM
To: Alpert, Reed; Openmama-dev@...
Subject: RE: Enforcing field type on publish

 

Hi Guy,

When you refer to 'addString' below you mention a cache, but this is not a method of the C/C++MamaFieldCache

are you refering to your own cache object or actually a MamaMsg object?

 

Regards,

Glenn 

 


From: openmama-dev-bounces@... [openmama-dev-bounces@...] on behalf of Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 13:59
To: Openmama-dev@...
Subject: [Openmama-dev] Enforcing field type on publish

Hi,

 

This is regarding a way to check that the field data type being published matches (closely) the dictionary data type.

 

The motivation for this is to prevent our internally published data caches from invalid data.

For example, if a field (name=MARK_PRICE fid=300) is a MamaPrice in the dictionary, and an app uses this:

                addString("MARK_PRICE", 300, "invalid price");

then we have a field that is now a string in the cache, and that causes problems with subscribing apps that use getPrice().

 

Currently we are looking to the bridges that we use (Tick42/TREP & Solace) to enforce these restrictions.

 

Adding this to the OpenMAMA layer is tricky, but allows a single policy to be enforced across multiple bridges. The main question from my point of view being how to set the policy. They can range from none (default), to most strict (published field's type must match dictionary), to common conversions supported (e.g., F32 -> F64), to a custom grid setting exactly what conversions are allowed.

 

I welcome any input from others who have run into this same issue, or found other ways to solve it.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.



_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


Benjamin Taieb
 

Hi Frank,

You right, I should specify further what we have in mind:

-Of course dictionary are not mandatory, and I don't think anyone wants the "dictionary check" to be default behaviour, it has to be optional to maintain compatibility with the many other use case. After all, mama is a middleware API, not a market data API and should be kept that way.

So property based behaviour here. In some respect, your point highlight also the fact that there is no way to prevent application developers from doing the "bad thing", as a property can always be changed. What we are after is to prevent applications developers from doing the "bad thing" by accident.

 

-UnitTest have been run and pass on our payload (well at least last time I checked), but last time I checked, they were no composite tests for scalar type for example.

The Qpid payload use a macro for all scalar, and in particular I believe:

pn_data_put##TYPE  (impl->mBody, value);

will result in "brutal" cast without any further check, i.e. truncation, interpreting double as I8 and all sort of nice things.

I don't think that is desirable from a design point of view, but would like to hear if people think there is a performance argument about it ?

Performance apart, I think also that defining behaviour in reference implementation is not desirable, comparing to having a specification. Just having a matrix somewhere (google spreadsheet or OpenMama wiki ?) will be very useful.

 

-That why the proposed changes is on the publisher, not on the subscriber, as in this case, there is strong asymmetry between publisher and subscriber, i.e. publisher are pretty low volume, but number of them, and subscribers will have higher traffic to process. I understand this is not the case for example with a low latency environment. That's why it is optional and not the default.

 

I agree about having flexibility in the cast belong to the payload, however I don't think UnitTest should verify the mandatory minimum, but rather enforces required interoperability, as this could makes payload incompatible. The whole benefit of using OpenMama collapse in this case in my opinion.

 

Cheers,

Ben.

 

 

From: Frank Quinn [mailto:fquinn@...]
Sent: 10 September 2014 09:31
To: Benjamin Taieb; Alpert, Reed; fquinn.ni@...; Glenn McClements
Cc: Openmama-dev@...
Subject: RE: [Openmama-dev] Enforcing field type on publish

 

Hi Ben,

 

There are a few things to consider here:

 

1.       The MAMA Dictionary is optional too. If you look at basic mamapublisherc / mamasubscriberc, you’ll see it doesn’t use the dictionary, but it does use mamaMsg.

2.       Unit tests actually do exist for the various mandatory castings required. If you run the payload unit tests today, for example, you’ll find that getting a date time from a string as well as a date time field should be supported, as well as other payload level casting expectations… In fact if you haven’t run these tests already, you really should, especially if you want to use MAMDA. If you look at the Qpid bridge, you’ll see a bunch of code put in to make this casting more forgiving.

3.       While iterating over a message, in the interest of performance, you probably don’t want to hit the data dictionary on every field being processed, particularly if you’re trying to do high performance field caching.

 

Basically, I’m of the opinion that if you want to make a payload flexible with respect to the field types it supports, that should be done at a payload layer and we have unit tests to verify the mandatory minimum.

 

If you want to enforce MAMA Dictionary compliance, this should be done in the MAMA application, where you can use field descriptors to verify content type at send and receive sides. E.g.

 

switch (mamaFieldDescriptor_getType (fieldDescriptor))

{

    case MAMA_FIELD_TYPE_I32:

    case MAMA_FIELD_TYPE_I64:

        allow the add / access;

    default:

        handle condition;

}

 

As well as everything else, whether or not the I64 to I32 casting (yes - the other way around) will be legal will depend on the data content being cast and sent across. Depending on the content, a U8 / U16 could even be fine. Without knowledge and consideration of this within the application, undesired truncation of data could be found, or opportunities to avoid failed data transfer due to strictness checking could be avoided.

 

Cheers,

Frank

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Benjamin Taieb
Sent: 10 September 2014 08:47
To: Alpert, Reed; fquinn.ni@...; Glenn McClements
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Just to put emphasis on what Reed explain:

It should be probably implemented in the C layer, making sure that it "catches" all languages "for free".

The goal is to enforce it in OpenMama layer, so that OpenMama application are forced to adheres to it.

 

I believe use of MamaFieldCache is optional, which would rules it out.

And referring to Glenn comments about mamaMsg, the goal is that the payload does not trust the calling application.

 

There is multiples challenges I believe in implementing that:

-How does the mama layer get dictionary ? This could be easily solves by using dictionnary requestor wherever we decides to implement it.

-How does the mapping between a particular msg and namespace is going to work ? At the moment, a dictionary is attached to namespace, namespace could be shared across multiples sources, but a message is source agnostic, only the publisher when it send the msg will send on a particular source/ symbol. On the other side, we would want to fail as soon as possible in the process, probably in the mamaMsg_addXX and mamaMsg_setXX calls, so that the calling application know for which fields it is doing something wrong.

-We need to clarify the "authorised cast": If you're doing addI32 on a I64 fields, should probably get authorised. At the moment, various Payload implement various level of flexibility, would be good that we defines UnitTests defining precisely what it should be, probably modelled from what is defined in wombatmsg, for retro-compatibility purpose.

 

I'll take any input from the community, particularly on point 2, which is the hardest in my opinion.

Cheers,

Ben.

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Alpert, Reed
Sent: 08 September 2014 20:10
To: fquinn.ni@...; Glenn McClements
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Yes, this is C++, but we also have Java and DotNet clients (some C# and some Excel).

We do publish best practices, but our app dev groups sometimes do not follow them, and we want to prevent the SolCache from getting out of sync with the dictionary.

We also provide them with sample apps that have this type of control, but sometimes … J

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Thursday, September 04, 2014 4:56 PM
To: Glenn McClements
Cc: Alpert, Reed; Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Guessing by your capitalization, I'm guessing this is C++? If so, have you considered mandating that all function calls use MamaFieldDescriptors pulled from a standardized dictionary rather than using string / fid pairs?

Cheers,
Frank

 

On Thu, Sep 4, 2014 at 9:33 PM, Glenn McClements <gmcclements@...> wrote:

There are a number of places where this *could* be done, but not all make sense to me:

 

- In the payload library - this doesn't feel right as I feel the payload should be 'dumb' and just trust the caller. Also it would need to be done for each supported payload. 

 

- In the payload bridge - same points as above

 

- In MamaMsg - this would work across all supported payloads, but again I think the payload should be trust the caller as to what fields should be what type and act a a simple wrapper 

 

- In a field cache (e.g. the C & C++ MamaFieldCache) - this makes the most sense to me as the logic for maintaining state about fields and if they have changed is already there, so extra rules and checks would belong here

 

With the current MamaFieldCache you could do this already if you primed the fields by creating them with the correct type on startup, this would depend on having a well known set of fields though because you don't want to be adding everything in the dictionary. You could also try this technique with the MamaMsg object, but it will be payload specific because some will throw an error if you try to update a field with a new type, and others may replace it.

 

Glenn 

 

 

 

From: Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 15:28
To: Glenn McClements; Openmama-dev@...


Subject: RE: Enforcing field type on publish

 

Hi,

 

Yes, the addString() is from MamaMsg.

 

The cache is our own market data infrastructure, not the MamaFieldCache.

In this case it is either Solace/SolCache or TREP RTIC.

 

We want business app dev groups to be able to look at the dictionary, see a field that is MamaPrice, and be guaranteed that MamaMsg.getPrice() will always work (although MamaPrice.getIsValidPrice() may return false).

A certain number of the type mismatches come from Excel, where a failed formula may populate a cell as #NA, and that gets published into a price/float field.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Wednesday, September 03, 2014 3:10 PM
To: Alpert, Reed; Openmama-dev@...
Subject: RE: Enforcing field type on publish

 

Hi Guy,

When you refer to 'addString' below you mention a cache, but this is not a method of the C/C++MamaFieldCache

are you refering to your own cache object or actually a MamaMsg object?

 

Regards,

Glenn 

 


From: openmama-dev-bounces@... [openmama-dev-bounces@...] on behalf of Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 13:59
To: Openmama-dev@...
Subject: [Openmama-dev] Enforcing field type on publish

Hi,

 

This is regarding a way to check that the field data type being published matches (closely) the dictionary data type.

 

The motivation for this is to prevent our internally published data caches from invalid data.

For example, if a field (name=MARK_PRICE fid=300) is a MamaPrice in the dictionary, and an app uses this:

                addString("MARK_PRICE", 300, "invalid price");

then we have a field that is now a string in the cache, and that causes problems with subscribing apps that use getPrice().

 

Currently we are looking to the bridges that we use (Tick42/TREP & Solace) to enforce these restrictions.

 

Adding this to the OpenMAMA layer is tricky, but allows a single policy to be enforced across multiple bridges. The main question from my point of view being how to set the policy. They can range from none (default), to most strict (published field's type must match dictionary), to common conversions supported (e.g., F32 -> F64), to a custom grid setting exactly what conversions are allowed.

 

I welcome any input from others who have run into this same issue, or found other ways to solve it.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.



_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.



Alpert, Reed <reed.alpert@...>
 

Hi,

 

Yes I agree with Ben’s comments.

Here are some caveats:

1.       This is a feature that is requested by our market data infrastructure team to prevent app dev teams from accidentally publishing incorrect field types. We don’t believe that a voluntary program will work, which is why the FieldCache as a location is not preferred. Apps won’t use the FieldCache if they don’t need it, or are using an Excel plugin for publishing (quite common).

2.       This is a configured feature, and only for publication.

3.       This is not for subscription, that is, a subscriber can use any of the getXX() methods that work.

 

There seems to be general agreement that the OM layer is much better place for this feature than bridges themselves.

 

The two best locations seem to be:

1.       In mamaMsg, in the addXX and updateXX methods, probably with a macro like CHECK_MODIFY() that calls out to another module to do the work.

a.       This is good since it keep the message construction in mamaMsg module.

b.      This is good since it give the programmer the quickest feedback on an error condition.

c.       This is bad since it puts some performance hit on construction of every message, even on subscribe.

d.      This is bad since it makes the code active for creating subscribe messages too.

e.      This is bad since it puts more code in more places for this feature.

2.       In publisher.send() that checks a config var and then calls out to another module to do the work.

a.       This is good since it has minimal code changes, and minimal performance effect.

b.      This is good since it only affects publish messages.

c.       This is bad since it gives delayed failure, and there may be multiple fields that fail.

                                                               i.      Probably logging each failed field and returning an error code will work well to help developers fix problems.

 

I think the publisher.send() method is the best place, given the above.

 

But figuring out which dictionary to use for the checking seems to be the most difficult part.

Adding this feature as a config option may require also adding some optional config info about how to get a dictionary for a specific bridge, e.g., the source name.

 

Once the dictionary loading/mapping is solved then the rest seems fairly straightforward, except for the determination of allowed/denied conversions as configurable policy, but starting with 3 simple ones seems best:

1.       No checking (default)

2.       Type must match dictionary.

3.       Type must match dictionary, with a small number of allowed conversions that do not incur data loss (e.g., F32 -> F64).

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Benjamin Taieb [mailto:benjamin.taieb@...]
Sent: Wednesday, September 10, 2014 6:06 AM
To: Frank Quinn; Alpert, Reed; fquinn.ni@...; Glenn McClements
Cc: Openmama-dev@...
Subject: RE: [Openmama-dev] Enforcing field type on publish

 

Hi Frank,

You right, I should specify further what we have in mind:

-Of course dictionary are not mandatory, and I don't think anyone wants the "dictionary check" to be default behaviour, it has to be optional to maintain compatibility with the many other use case. After all, mama is a middleware API, not a market data API and should be kept that way.

So property based behaviour here. In some respect, your point highlight also the fact that there is no way to prevent application developers from doing the "bad thing", as a property can always be changed. What we are after is to prevent applications developers from doing the "bad thing" by accident.

 

-UnitTest have been run and pass on our payload (well at least last time I checked), but last time I checked, they were no composite tests for scalar type for example.

The Qpid payload use a macro for all scalar, and in particular I believe:

pn_data_put##TYPE  (impl->mBody, value);

will result in "brutal" cast without any further check, i.e. truncation, interpreting double as I8 and all sort of nice things.

I don't think that is desirable from a design point of view, but would like to hear if people think there is a performance argument about it ?

Performance apart, I think also that defining behaviour in reference implementation is not desirable, comparing to having a specification. Just having a matrix somewhere (google spreadsheet or OpenMama wiki ?) will be very useful.

 

-That why the proposed changes is on the publisher, not on the subscriber, as in this case, there is strong asymmetry between publisher and subscriber, i.e. publisher are pretty low volume, but number of them, and subscribers will have higher traffic to process. I understand this is not the case for example with a low latency environment. That's why it is optional and not the default.

 

I agree about having flexibility in the cast belong to the payload, however I don't think UnitTest should verify the mandatory minimum, but rather enforces required interoperability, as this could makes payload incompatible. The whole benefit of using OpenMama collapse in this case in my opinion.

 

Cheers,

Ben.

 

 

 

From: Frank Quinn [mailto:fquinn@...]
Sent: 10 September 2014 09:31
To: Benjamin Taieb; Alpert, Reed; fquinn.ni@...; Glenn McClements
Cc: Openmama-dev@...
Subject: RE: [Openmama-dev] Enforcing field type on publish

 

Hi Ben,

 

There are a few things to consider here:

 

1.       The MAMA Dictionary is optional too. If you look at basic mamapublisherc / mamasubscriberc, you’ll see it doesn’t use the dictionary, but it does use mamaMsg.

2.       Unit tests actually do exist for the various mandatory castings required. If you run the payload unit tests today, for example, you’ll find that getting a date time from a string as well as a date time field should be supported, as well as other payload level casting expectations… In fact if you haven’t run these tests already, you really should, especially if you want to use MAMDA. If you look at the Qpid bridge, you’ll see a bunch of code put in to make this casting more forgiving.

3.       While iterating over a message, in the interest of performance, you probably don’t want to hit the data dictionary on every field being processed, particularly if you’re trying to do high performance field caching.

 

Basically, I’m of the opinion that if you want to make a payload flexible with respect to the field types it supports, that should be done at a payload layer and we have unit tests to verify the mandatory minimum.

 

If you want to enforce MAMA Dictionary compliance, this should be done in the MAMA application, where you can use field descriptors to verify content type at send and receive sides. E.g.

 

switch (mamaFieldDescriptor_getType (fieldDescriptor))

{

    case MAMA_FIELD_TYPE_I32:

    case MAMA_FIELD_TYPE_I64:

        allow the add / access;

    default:

        handle condition;

}

 

As well as everything else, whether or not the I64 to I32 casting (yes - the other way around) will be legal will depend on the data content being cast and sent across. Depending on the content, a U8 / U16 could even be fine. Without knowledge and consideration of this within the application, undesired truncation of data could be found, or opportunities to avoid failed data transfer due to strictness checking could be avoided.

 

Cheers,

Frank

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Benjamin Taieb
Sent: 10 September 2014 08:47
To: Alpert, Reed; fquinn.ni@...; Glenn McClements
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Just to put emphasis on what Reed explain:

It should be probably implemented in the C layer, making sure that it "catches" all languages "for free".

The goal is to enforce it in OpenMama layer, so that OpenMama application are forced to adheres to it.

 

I believe use of MamaFieldCache is optional, which would rules it out.

And referring to Glenn comments about mamaMsg, the goal is that the payload does not trust the calling application.

 

There is multiples challenges I believe in implementing that:

-How does the mama layer get dictionary ? This could be easily solves by using dictionnary requestor wherever we decides to implement it.

-How does the mapping between a particular msg and namespace is going to work ? At the moment, a dictionary is attached to namespace, namespace could be shared across multiples sources, but a message is source agnostic, only the publisher when it send the msg will send on a particular source/ symbol. On the other side, we would want to fail as soon as possible in the process, probably in the mamaMsg_addXX and mamaMsg_setXX calls, so that the calling application know for which fields it is doing something wrong.

-We need to clarify the "authorised cast": If you're doing addI32 on a I64 fields, should probably get authorised. At the moment, various Payload implement various level of flexibility, would be good that we defines UnitTests defining precisely what it should be, probably modelled from what is defined in wombatmsg, for retro-compatibility purpose.

 

I'll take any input from the community, particularly on point 2, which is the hardest in my opinion.

Cheers,

Ben.

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Alpert, Reed
Sent: 08 September 2014 20:10
To: fquinn.ni@...; Glenn McClements
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Yes, this is C++, but we also have Java and DotNet clients (some C# and some Excel).

We do publish best practices, but our app dev groups sometimes do not follow them, and we want to prevent the SolCache from getting out of sync with the dictionary.

We also provide them with sample apps that have this type of control, but sometimes … J

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Thursday, September 04, 2014 4:56 PM
To: Glenn McClements
Cc: Alpert, Reed; Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Guessing by your capitalization, I'm guessing this is C++? If so, have you considered mandating that all function calls use MamaFieldDescriptors pulled from a standardized dictionary rather than using string / fid pairs?

Cheers,
Frank

 

On Thu, Sep 4, 2014 at 9:33 PM, Glenn McClements <gmcclements@...> wrote:

There are a number of places where this *could* be done, but not all make sense to me:

 

- In the payload library - this doesn't feel right as I feel the payload should be 'dumb' and just trust the caller. Also it would need to be done for each supported payload. 

 

- In the payload bridge - same points as above

 

- In MamaMsg - this would work across all supported payloads, but again I think the payload should be trust the caller as to what fields should be what type and act a a simple wrapper 

 

- In a field cache (e.g. the C & C++ MamaFieldCache) - this makes the most sense to me as the logic for maintaining state about fields and if they have changed is already there, so extra rules and checks would belong here

 

With the current MamaFieldCache you could do this already if you primed the fields by creating them with the correct type on startup, this would depend on having a well known set of fields though because you don't want to be adding everything in the dictionary. You could also try this technique with the MamaMsg object, but it will be payload specific because some will throw an error if you try to update a field with a new type, and others may replace it.

 

Glenn 

 

 

 

From: Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 15:28
To: Glenn McClements; Openmama-dev@...


Subject: RE: Enforcing field type on publish

 

Hi,

 

Yes, the addString() is from MamaMsg.

 

The cache is our own market data infrastructure, not the MamaFieldCache.

In this case it is either Solace/SolCache or TREP RTIC.

 

We want business app dev groups to be able to look at the dictionary, see a field that is MamaPrice, and be guaranteed that MamaMsg.getPrice() will always work (although MamaPrice.getIsValidPrice() may return false).

A certain number of the type mismatches come from Excel, where a failed formula may populate a cell as #NA, and that gets published into a price/float field.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Wednesday, September 03, 2014 3:10 PM
To: Alpert, Reed; Openmama-dev@...
Subject: RE: Enforcing field type on publish

 

Hi Guy,

When you refer to 'addString' below you mention a cache, but this is not a method of the C/C++MamaFieldCache

are you refering to your own cache object or actually a MamaMsg object?

 

Regards,

Glenn 

 


From: openmama-dev-bounces@... [openmama-dev-bounces@...] on behalf of Alpert, Reed [reed.alpert@...]
Sent: 03 September 2014 13:59
To: Openmama-dev@...
Subject: [Openmama-dev] Enforcing field type on publish

Hi,

 

This is regarding a way to check that the field data type being published matches (closely) the dictionary data type.

 

The motivation for this is to prevent our internally published data caches from invalid data.

For example, if a field (name=MARK_PRICE fid=300) is a MamaPrice in the dictionary, and an app uses this:

                addString("MARK_PRICE", 300, "invalid price");

then we have a field that is now a string in the cache, and that causes problems with subscribing apps that use getPrice().

 

Currently we are looking to the bridges that we use (Tick42/TREP & Solace) to enforce these restrictions.

 

Adding this to the OpenMAMA layer is tricky, but allows a single policy to be enforced across multiple bridges. The main question from my point of view being how to set the policy. They can range from none (default), to most strict (published field's type must match dictionary), to common conversions supported (e.g., F32 -> F64), to a custom grid setting exactly what conversions are allowed.

 

I welcome any input from others who have run into this same issue, or found other ways to solve it.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.



_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Glenn McClements <gmcclements@...>
 

Hi Alpert, 
I like #2 more, but I still want to keep the more data model/market data/custom logic away from the core messaging part of OpenMAMA. 

What I’m thinking of is a hook/callback to a programmer defined function that would take the publisher, transport, message, a user defined closure etc that would return a status to indicate if the message should be published or not. The dictionary etc could be obtained at initialisation time through a separate init function. 

This dovetails with some of the other pieces of work going on:
  • we now have a framework for loading bridges dynamically, so the functionality could be in a library loaded automatically at startup
  • the hook above looks very much like publisher side entitlements (pass a publisher, message and expect a yes or no back) so we do 

I still need to work though this a bit more but let me know what you think.

Glenn 

From: <Alpert>, Reed <reed.alpert@...>
Date: Wednesday, 10 September 2014 19:13
To: Benjamin Taieb <benjamin.taieb@...>, Frank Quinn <fquinn@...>, "fquinn.ni@..." <fquinn.ni@...>, Glenn McClements <gmcclements@...>
Cc: "Openmama-dev@..." <Openmama-dev@...>
Subject: RE: [Openmama-dev] Enforcing field type on publish

Hi,

 

Yes I agree with Ben’s comments.

Here are some caveats:

1.       This is a feature that is requested by our market data infrastructure team to prevent app dev teams from accidentally publishing incorrect field types. We don’t believe that a voluntary program will work, which is why the FieldCache as a location is not preferred. Apps won’t use the FieldCache if they don’t need it, or are using an Excel plugin for publishing (quite common).

2.       This is a configured feature, and only for publication.

3.       This is not for subscription, that is, a subscriber can use any of the getXX() methods that work.

 

There seems to be general agreement that the OM layer is much better place for this feature than bridges themselves.

 

The two best locations seem to be:

1.       In mamaMsg, in the addXX and updateXX methods, probably with a macro like CHECK_MODIFY() that calls out to another module to do the work.

a.       This is good since it keep the message construction in mamaMsg module.

b.      This is good since it give the programmer the quickest feedback on an error condition.

c.       This is bad since it puts some performance hit on construction of every message, even on subscribe.

d.      This is bad since it makes the code active for creating subscribe messages too.

e.      This is bad since it puts more code in more places for this feature.

2.       In publisher.send() that checks a config var and then calls out to another module to do the work.

a.       This is good since it has minimal code changes, and minimal performance effect.

b.      This is good since it only affects publish messages.

c.       This is bad since it gives delayed failure, and there may be multiple fields that fail.

                                                               i.      Probably logging each failed field and returning an error code will work well to help developers fix problems.

 

I think the publisher.send() method is the best place, given the above.

 

But figuring out which dictionary to use for the checking seems to be the most difficult part.

Adding this feature as a config option may require also adding some optional config info about how to get a dictionary for a specific bridge, e.g., the source name.

 

Once the dictionary loading/mapping is solved then the rest seems fairly straightforward, except for the determination of allowed/denied conversions as configurable policy, but starting with 3 simple ones seems best:

1.       No checking (default)

2.       Type must match dictionary.

3.       Type must match dictionary, with a small number of allowed conversions that do not incur data loss (e.g., F32 -> F64).

 

Thanks,

 

Reed.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


Alpert, Reed <reed.alpert@...>
 

Hi,

 

I agree, the callback to a module that can do publish checking/etc is good.

 

I prototyped a checking module, and it works, although loading the dictionary on 1st publish has a small delay, and also required that the publish not be from a mama timer callback, since that deadlocked either on sending the subscription on the throttle queue, or getting the dict back.

 

We actually have more requirements for publish that are very specific to our env (adding system audit fields to all publishes to track when/where the data comes from).

Having a callback we can engage would allow us to do this w/o any other users needing to have our stuff in the way.

So loading our module at runtime and calling it would work quite well.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Thursday, September 11, 2014 5:20 AM
To: Alpert, Reed; Benjamin Taieb; Frank Quinn; fquinn.ni@...
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Hi Alpert, 

I like #2 more, but I still want to keep the more data model/market data/custom logic away from the core messaging part of OpenMAMA. 

 

What I’m thinking of is a hook/callback to a programmer defined function that would take the publisher, transport, message, a user defined closure etc that would return a status to indicate if the message should be published or not. The dictionary etc could be obtained at initialisation time through a separate init function. 

 

This dovetails with some of the other pieces of work going on:

  • we now have a framework for loading bridges dynamically, so the functionality could be in a library loaded automatically at startup
  • the hook above looks very much like publisher side entitlements (pass a publisher, message and expect a yes or no back) so we do 

 

I still need to work though this a bit more but let me know what you think.

 

Glenn 

 

From: <Alpert>, Reed <reed.alpert@...>
Date: Wednesday, 10 September 2014 19:13
To: Benjamin Taieb <benjamin.taieb@...>, Frank Quinn <fquinn@...>, "fquinn.ni@..." <fquinn.ni@...>, Glenn McClements <gmcclements@...>
Cc: "Openmama-dev@..." <Openmama-dev@...>
Subject: RE: [Openmama-dev] Enforcing field type on publish

 

Hi,

 

Yes I agree with Ben’s comments.

Here are some caveats:

1.      This is a feature that is requested by our market data infrastructure team to prevent app dev teams from accidentally publishing incorrect field types. We don’t believe that a voluntary program will work, which is why the FieldCache as a location is not preferred. Apps won’t use the FieldCache if they don’t need it, or are using an Excel plugin for publishing (quite common).

2.      This is a configured feature, and only for publication.

3.      This is not for subscription, that is, a subscriber can use any of the getXX() methods that work.

 

There seems to be general agreement that the OM layer is much better place for this feature than bridges themselves.

 

The two best locations seem to be:

1.      In mamaMsg, in the addXX and updateXX methods, probably with a macro like CHECK_MODIFY() that calls out to another module to do the work.

a.       This is good since it keep the message construction in mamaMsg module.

b.      This is good since it give the programmer the quickest feedback on an error condition.

c.       This is bad since it puts some performance hit on construction of every message, even on subscribe.

d.      This is bad since it makes the code active for creating subscribe messages too.

e.       This is bad since it puts more code in more places for this feature.

2.      In publisher.send() that checks a config var and then calls out to another module to do the work.

a.       This is good since it has minimal code changes, and minimal performance effect.

b.      This is good since it only affects publish messages.

c.       This is bad since it gives delayed failure, and there may be multiple fields that fail.

                                                              i.      Probably logging each failed field and returning an error code will work well to help developers fix problems.

 

I think the publisher.send() method is the best place, given the above.

 

But figuring out which dictionary to use for the checking seems to be the most difficult part.

Adding this feature as a config option may require also adding some optional config info about how to get a dictionary for a specific bridge, e.g., the source name.

 

Once the dictionary loading/mapping is solved then the rest seems fairly straightforward, except for the determination of allowed/denied conversions as configurable policy, but starting with 3 simple ones seems best:

1.      No checking (default)

2.      Type must match dictionary.

3.      Type must match dictionary, with a small number of allowed conversions that do not incur data loss (e.g., F32 -> F64).

 

Thanks,

 

Reed.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Damian Maguire
 

Hey Reed, 

This all sounds really interesting, particularly the fact that you have already identified multiple use cases for publish side hooks. A lot of it actually ties very nicely with some of the internal discussions we've been having around a plugin architecture for OpenMAMA as well, which in turn sit quite nicely with the library manager work which is also ongoing. 

At present myself and the team here are pretty solidly booked with a few internal projects (which should provide some real benefits to OpenMAMA when they're complete), so it'll be a few weeks before we can sit down and tackle this properly. However, I'd be very keen to get some full community engagement in discussing this, and work to put together a design that covers the requirements of the majority of potential use cases. With that in mind, if you guys can think of anywhere else where you believe hooks would be useful, or if you can think of any specific requirements for your current ideas, feel free to send them over and we'll start collating everything. 

Thanks, 

Damian



On Wed, Sep 17, 2014 at 8:03 PM, Alpert, Reed <reed.alpert@...> wrote:

Hi,

 

I agree, the callback to a module that can do publish checking/etc is good.

 

I prototyped a checking module, and it works, although loading the dictionary on 1st publish has a small delay, and also required that the publish not be from a mama timer callback, since that deadlocked either on sending the subscription on the throttle queue, or getting the dict back.

 

We actually have more requirements for publish that are very specific to our env (adding system audit fields to all publishes to track when/where the data comes from).

Having a callback we can engage would allow us to do this w/o any other users needing to have our stuff in the way.

So loading our module at runtime and calling it would work quite well.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Thursday, September 11, 2014 5:20 AM
To: Alpert, Reed; Benjamin Taieb; Frank Quinn; fquinn.ni@...
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Hi Alpert, 

I like #2 more, but I still want to keep the more data model/market data/custom logic away from the core messaging part of OpenMAMA. 

 

What I’m thinking of is a hook/callback to a programmer defined function that would take the publisher, transport, message, a user defined closure etc that would return a status to indicate if the message should be published or not. The dictionary etc could be obtained at initialisation time through a separate init function. 

 

This dovetails with some of the other pieces of work going on:

  • we now have a framework for loading bridges dynamically, so the functionality could be in a library loaded automatically at startup
  • the hook above looks very much like publisher side entitlements (pass a publisher, message and expect a yes or no back) so we do 

 

I still need to work though this a bit more but let me know what you think.

 

Glenn 

 

From: <Alpert>, Reed <reed.alpert@...>
Date: Wednesday, 10 September 2014 19:13
To: Benjamin Taieb <benjamin.taieb@...>, Frank Quinn <fquinn@...>, "fquinn.ni@..." <fquinn.ni@...>, Glenn McClements <gmcclements@...>
Cc: "Openmama-dev@..." <Openmama-dev@...>
Subject: RE: [Openmama-dev] Enforcing field type on publish

 

Hi,

 

Yes I agree with Ben’s comments.

Here are some caveats:

1.      This is a feature that is requested by our market data infrastructure team to prevent app dev teams from accidentally publishing incorrect field types. We don’t believe that a voluntary program will work, which is why the FieldCache as a location is not preferred. Apps won’t use the FieldCache if they don’t need it, or are using an Excel plugin for publishing (quite common).

2.      This is a configured feature, and only for publication.

3.      This is not for subscription, that is, a subscriber can use any of the getXX() methods that work.

 

There seems to be general agreement that the OM layer is much better place for this feature than bridges themselves.

 

The two best locations seem to be:

1.      In mamaMsg, in the addXX and updateXX methods, probably with a macro like CHECK_MODIFY() that calls out to another module to do the work.

a.       This is good since it keep the message construction in mamaMsg module.

b.      This is good since it give the programmer the quickest feedback on an error condition.

c.       This is bad since it puts some performance hit on construction of every message, even on subscribe.

d.      This is bad since it makes the code active for creating subscribe messages too.

e.       This is bad since it puts more code in more places for this feature.

2.      In publisher.send() that checks a config var and then calls out to another module to do the work.

a.       This is good since it has minimal code changes, and minimal performance effect.

b.      This is good since it only affects publish messages.

c.       This is bad since it gives delayed failure, and there may be multiple fields that fail.

                                                              i.      Probably logging each failed field and returning an error code will work well to help developers fix problems.

 

I think the publisher.send() method is the best place, given the above.

 

But figuring out which dictionary to use for the checking seems to be the most difficult part.

Adding this feature as a config option may require also adding some optional config info about how to get a dictionary for a specific bridge, e.g., the source name.

 

Once the dictionary loading/mapping is solved then the rest seems fairly straightforward, except for the determination of allowed/denied conversions as configurable policy, but starting with 3 simple ones seems best:

1.      No checking (default)

2.      Type must match dictionary.

3.      Type must match dictionary, with a small number of allowed conversions that do not incur data loss (e.g., F32 -> F64).

 

Thanks,

 

Reed.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev



Alpert, Reed <reed.alpert@...>
 

Hi Damian,

 

Yes, having a pluggable module that handles some pre/post events makes sense.

Our handler for this now is very JPM-specific, and I don’t think it would make sense to try and standardize that (checks field types, ensures there is a msgType, and adds some audit fields to track who/when/where published the msg).

 

To get a dictionary I put a hook in at the end of the transport create, since that is the first time we have enough info to check the config in mama.properties and then subscribe to the dictionary.

If that is OK then the dictionary is stashed in the transport data, and the pre-publish looks at that to see if it should apply the checks.

 

Begin able to load a shared obj via the library manager would work great.

For the pre/post events it seems like a lot to have a general list like transport create and publish and maybe bridge create, etc.

I’m not sure if that is overkill to fully fill out the list of pre/post events across many objects.

 

To store library data in the main objects’ data (e.g., bridge, transport, publisher, etc) maybe a user closure can be added.

Of course that assumes that a single pre/post library is installed.

 

I’ll keep adding things as our development continue.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Damian Maguire [mailto:damian@...]
Sent: Friday, September 19, 2014 11:53 AM
To: Alpert, Reed
Cc: Glenn McClements; Benjamin Taieb; Frank Quinn; fquinn.ni@...; Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Hey Reed, 

 

This all sounds really interesting, particularly the fact that you have already identified multiple use cases for publish side hooks. A lot of it actually ties very nicely with some of the internal discussions we've been having around a plugin architecture for OpenMAMA as well, which in turn sit quite nicely with the library manager work which is also ongoing. 

 

At present myself and the team here are pretty solidly booked with a few internal projects (which should provide some real benefits to OpenMAMA when they're complete), so it'll be a few weeks before we can sit down and tackle this properly. However, I'd be very keen to get some full community engagement in discussing this, and work to put together a design that covers the requirements of the majority of potential use cases. With that in mind, if you guys can think of anywhere else where you believe hooks would be useful, or if you can think of any specific requirements for your current ideas, feel free to send them over and we'll start collating everything. 

 

Thanks, 

 

Damian

 

 

 

On Wed, Sep 17, 2014 at 8:03 PM, Alpert, Reed <reed.alpert@...> wrote:

Hi,

 

I agree, the callback to a module that can do publish checking/etc is good.

 

I prototyped a checking module, and it works, although loading the dictionary on 1st publish has a small delay, and also required that the publish not be from a mama timer callback, since that deadlocked either on sending the subscription on the throttle queue, or getting the dict back.

 

We actually have more requirements for publish that are very specific to our env (adding system audit fields to all publishes to track when/where the data comes from).

Having a callback we can engage would allow us to do this w/o any other users needing to have our stuff in the way.

So loading our module at runtime and calling it would work quite well.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Thursday, September 11, 2014 5:20 AM
To: Alpert, Reed; Benjamin Taieb; Frank Quinn; fquinn.ni@...
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Hi Alpert, 

I like #2 more, but I still want to keep the more data model/market data/custom logic away from the core messaging part of OpenMAMA. 

 

What I’m thinking of is a hook/callback to a programmer defined function that would take the publisher, transport, message, a user defined closure etc that would return a status to indicate if the message should be published or not. The dictionary etc could be obtained at initialisation time through a separate init function. 

 

This dovetails with some of the other pieces of work going on:

  • we now have a framework for loading bridges dynamically, so the functionality could be in a library loaded automatically at startup
  • the hook above looks very much like publisher side entitlements (pass a publisher, message and expect a yes or no back) so we do 

 

I still need to work though this a bit more but let me know what you think.

 

Glenn 

 

From: <Alpert>, Reed <reed.alpert@...>
Date: Wednesday, 10 September 2014 19:13
To: Benjamin Taieb <benjamin.taieb@...>, Frank Quinn <fquinn@...>, "fquinn.ni@..." <fquinn.ni@...>, Glenn McClements <gmcclements@...>
Cc: "Openmama-dev@..." <Openmama-dev@...>
Subject: RE: [Openmama-dev] Enforcing field type on publish

 

Hi,

 

Yes I agree with Ben’s comments.

Here are some caveats:

1.      This is a feature that is requested by our market data infrastructure team to prevent app dev teams from accidentally publishing incorrect field types. We don’t believe that a voluntary program will work, which is why the FieldCache as a location is not preferred. Apps won’t use the FieldCache if they don’t need it, or are using an Excel plugin for publishing (quite common).

2.      This is a configured feature, and only for publication.

3.      This is not for subscription, that is, a subscriber can use any of the getXX() methods that work.

 

There seems to be general agreement that the OM layer is much better place for this feature than bridges themselves.

 

The two best locations seem to be:

1.      In mamaMsg, in the addXX and updateXX methods, probably with a macro like CHECK_MODIFY() that calls out to another module to do the work.

a.       This is good since it keep the message construction in mamaMsg module.

b.      This is good since it give the programmer the quickest feedback on an error condition.

c.       This is bad since it puts some performance hit on construction of every message, even on subscribe.

d.      This is bad since it makes the code active for creating subscribe messages too.

e.       This is bad since it puts more code in more places for this feature.

2.      In publisher.send() that checks a config var and then calls out to another module to do the work.

a.       This is good since it has minimal code changes, and minimal performance effect.

b.      This is good since it only affects publish messages.

c.       This is bad since it gives delayed failure, and there may be multiple fields that fail.

                                                              i.      Probably logging each failed field and returning an error code will work well to help developers fix problems.

 

I think the publisher.send() method is the best place, given the above.

 

But figuring out which dictionary to use for the checking seems to be the most difficult part.

Adding this feature as a config option may require also adding some optional config info about how to get a dictionary for a specific bridge, e.g., the source name.

 

Once the dictionary loading/mapping is solved then the rest seems fairly straightforward, except for the determination of allowed/denied conversions as configurable policy, but starting with 3 simple ones seems best:

1.      No checking (default)

2.      Type must match dictionary.

3.      Type must match dictionary, with a small number of allowed conversions that do not incur data loss (e.g., F32 -> F64).

 

Thanks,

 

Reed.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Alpert, Reed <reed.alpert@...>
 

Hi Damian,

 

I have our pre-publish check code ready, but it is not pluggable.

I’d like to submit it as a patch using #ifdef like the entitlements code.

That way I don’t have to maintain it as OM evolves, and it won’t affect other OM developers.

When the pluggable/hook system is ready I can port it over.

 

There are 2 new files:

mama/c_cpp/src/c/publisherfieldcheck.c

mama/c_cpp/src/c/publisherfieldcheck.h

 

and changes to:

mama/c_cpp/src/c/Sconscript

mama/c_cpp/src/c/transport.c

mama/c_cpp/src/c/mama/transport.h

mama/c_cpp/src/c/publisher.c

 

Please let me know if an #ifdef patch is OK for this.

I can submit the patch if you want to see what it involves.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Alpert, Reed
Sent: Monday, September 22, 2014 6:17 PM
To: 'Damian Maguire'
Cc: Glenn McClements; Benjamin Taieb; Frank Quinn; fquinn.ni@...; Openmama-dev@...
Subject: RE: [Openmama-dev] Enforcing field type on publish

 

Hi Damian,

 

Yes, having a pluggable module that handles some pre/post events makes sense.

Our handler for this now is very JPM-specific, and I don’t think it would make sense to try and standardize that (checks field types, ensures there is a msgType, and adds some audit fields to track who/when/where published the msg).

 

To get a dictionary I put a hook in at the end of the transport create, since that is the first time we have enough info to check the config in mama.properties and then subscribe to the dictionary.

If that is OK then the dictionary is stashed in the transport data, and the pre-publish looks at that to see if it should apply the checks.

 

Begin able to load a shared obj via the library manager would work great.

For the pre/post events it seems like a lot to have a general list like transport create and publish and maybe bridge create, etc.

I’m not sure if that is overkill to fully fill out the list of pre/post events across many objects.

 

To store library data in the main objects’ data (e.g., bridge, transport, publisher, etc) maybe a user closure can be added.

Of course that assumes that a single pre/post library is installed.

 

I’ll keep adding things as our development continue.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Damian Maguire [mailto:damian@...]
Sent: Friday, September 19, 2014 11:53 AM
To: Alpert, Reed
Cc: Glenn McClements; Benjamin Taieb; Frank Quinn; fquinn.ni@...; Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Hey Reed, 

 

This all sounds really interesting, particularly the fact that you have already identified multiple use cases for publish side hooks. A lot of it actually ties very nicely with some of the internal discussions we've been having around a plugin architecture for OpenMAMA as well, which in turn sit quite nicely with the library manager work which is also ongoing. 

 

At present myself and the team here are pretty solidly booked with a few internal projects (which should provide some real benefits to OpenMAMA when they're complete), so it'll be a few weeks before we can sit down and tackle this properly. However, I'd be very keen to get some full community engagement in discussing this, and work to put together a design that covers the requirements of the majority of potential use cases. With that in mind, if you guys can think of anywhere else where you believe hooks would be useful, or if you can think of any specific requirements for your current ideas, feel free to send them over and we'll start collating everything. 

 

Thanks, 

 

Damian

 

 

 

On Wed, Sep 17, 2014 at 8:03 PM, Alpert, Reed <reed.alpert@...> wrote:

Hi,

 

I agree, the callback to a module that can do publish checking/etc is good.

 

I prototyped a checking module, and it works, although loading the dictionary on 1st publish has a small delay, and also required that the publish not be from a mama timer callback, since that deadlocked either on sending the subscription on the throttle queue, or getting the dict back.

 

We actually have more requirements for publish that are very specific to our env (adding system audit fields to all publishes to track when/where the data comes from).

Having a callback we can engage would allow us to do this w/o any other users needing to have our stuff in the way.

So loading our module at runtime and calling it would work quite well.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Thursday, September 11, 2014 5:20 AM
To: Alpert, Reed; Benjamin Taieb; Frank Quinn; fquinn.ni@...
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Hi Alpert, 

I like #2 more, but I still want to keep the more data model/market data/custom logic away from the core messaging part of OpenMAMA. 

 

What I’m thinking of is a hook/callback to a programmer defined function that would take the publisher, transport, message, a user defined closure etc that would return a status to indicate if the message should be published or not. The dictionary etc could be obtained at initialisation time through a separate init function. 

 

This dovetails with some of the other pieces of work going on:

  • we now have a framework for loading bridges dynamically, so the functionality could be in a library loaded automatically at startup
  • the hook above looks very much like publisher side entitlements (pass a publisher, message and expect a yes or no back) so we do 

 

I still need to work though this a bit more but let me know what you think.

 

Glenn 

 

From: <Alpert>, Reed <reed.alpert@...>
Date: Wednesday, 10 September 2014 19:13
To: Benjamin Taieb <benjamin.taieb@...>, Frank Quinn <fquinn@...>, "fquinn.ni@..." <fquinn.ni@...>, Glenn McClements <gmcclements@...>
Cc: "Openmama-dev@..." <Openmama-dev@...>
Subject: RE: [Openmama-dev] Enforcing field type on publish

 

Hi,

 

Yes I agree with Ben’s comments.

Here are some caveats:

1.      This is a feature that is requested by our market data infrastructure team to prevent app dev teams from accidentally publishing incorrect field types. We don’t believe that a voluntary program will work, which is why the FieldCache as a location is not preferred. Apps won’t use the FieldCache if they don’t need it, or are using an Excel plugin for publishing (quite common).

2.      This is a configured feature, and only for publication.

3.      This is not for subscription, that is, a subscriber can use any of the getXX() methods that work.

 

There seems to be general agreement that the OM layer is much better place for this feature than bridges themselves.

 

The two best locations seem to be:

1.      In mamaMsg, in the addXX and updateXX methods, probably with a macro like CHECK_MODIFY() that calls out to another module to do the work.

a.       This is good since it keep the message construction in mamaMsg module.

b.      This is good since it give the programmer the quickest feedback on an error condition.

c.       This is bad since it puts some performance hit on construction of every message, even on subscribe.

d.      This is bad since it makes the code active for creating subscribe messages too.

e.       This is bad since it puts more code in more places for this feature.

2.      In publisher.send() that checks a config var and then calls out to another module to do the work.

a.       This is good since it has minimal code changes, and minimal performance effect.

b.      This is good since it only affects publish messages.

c.       This is bad since it gives delayed failure, and there may be multiple fields that fail.

                                                              i.      Probably logging each failed field and returning an error code will work well to help developers fix problems.

 

I think the publisher.send() method is the best place, given the above.

 

But figuring out which dictionary to use for the checking seems to be the most difficult part.

Adding this feature as a config option may require also adding some optional config info about how to get a dictionary for a specific bridge, e.g., the source name.

 

Once the dictionary loading/mapping is solved then the rest seems fairly straightforward, except for the determination of allowed/denied conversions as configurable policy, but starting with 3 simple ones seems best:

1.      No checking (default)

2.      Type must match dictionary.

3.      Type must match dictionary, with a small number of allowed conversions that do not incur data loss (e.g., F32 -> F64).

 

Thanks,

 

Reed.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


Gary Molloy <g.molloy@...>
 

Hi Reed,

 

Thanks for getting back to us.

 

I think, as you say, the goal will be to have this feature as a plugin and not as an #ifdef, but we would be very interested in seeing your current implementation with an #ifdef patch to keep the discussion going on this.

 

Thanks,

Gary

 

 

 

Gary Molloy

SR.LABS Proven High Speed Electronic Trading Solutions

g.molloy@...

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Alpert, Reed
Sent: 18 November 2014 15:10
To: Alpert, Reed; Damian Maguire
Cc: Openmama-dev@...; Glenn McClements; Frank Quinn
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Hi Damian,

 

I have our pre-publish check code ready, but it is not pluggable.

I’d like to submit it as a patch using #ifdef like the entitlements code.

That way I don’t have to maintain it as OM evolves, and it won’t affect other OM developers.

When the pluggable/hook system is ready I can port it over.

 

There are 2 new files:

mama/c_cpp/src/c/publisherfieldcheck.c

mama/c_cpp/src/c/publisherfieldcheck.h

 

and changes to:

mama/c_cpp/src/c/Sconscript

mama/c_cpp/src/c/transport.c

mama/c_cpp/src/c/mama/transport.h

mama/c_cpp/src/c/publisher.c

 

Please let me know if an #ifdef patch is OK for this.

I can submit the patch if you want to see what it involves.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Alpert, Reed
Sent: Monday, September 22, 2014 6:17 PM
To: 'Damian Maguire'
Cc: Glenn McClements; Benjamin Taieb; Frank Quinn; fquinn.ni@...; Openmama-dev@...
Subject: RE: [Openmama-dev] Enforcing field type on publish

 

Hi Damian,

 

Yes, having a pluggable module that handles some pre/post events makes sense.

Our handler for this now is very JPM-specific, and I don’t think it would make sense to try and standardize that (checks field types, ensures there is a msgType, and adds some audit fields to track who/when/where published the msg).

 

To get a dictionary I put a hook in at the end of the transport create, since that is the first time we have enough info to check the config in mama.properties and then subscribe to the dictionary.

If that is OK then the dictionary is stashed in the transport data, and the pre-publish looks at that to see if it should apply the checks.

 

Begin able to load a shared obj via the library manager would work great.

For the pre/post events it seems like a lot to have a general list like transport create and publish and maybe bridge create, etc.

I’m not sure if that is overkill to fully fill out the list of pre/post events across many objects.

 

To store library data in the main objects’ data (e.g., bridge, transport, publisher, etc) maybe a user closure can be added.

Of course that assumes that a single pre/post library is installed.

 

I’ll keep adding things as our development continue.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Damian Maguire [mailto:damian@...]
Sent: Friday, September 19, 2014 11:53 AM
To: Alpert, Reed
Cc: Glenn McClements; Benjamin Taieb; Frank Quinn; fquinn.ni@...; Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Hey Reed, 

 

This all sounds really interesting, particularly the fact that you have already identified multiple use cases for publish side hooks. A lot of it actually ties very nicely with some of the internal discussions we've been having around a plugin architecture for OpenMAMA as well, which in turn sit quite nicely with the library manager work which is also ongoing. 

 

At present myself and the team here are pretty solidly booked with a few internal projects (which should provide some real benefits to OpenMAMA when they're complete), so it'll be a few weeks before we can sit down and tackle this properly. However, I'd be very keen to get some full community engagement in discussing this, and work to put together a design that covers the requirements of the majority of potential use cases. With that in mind, if you guys can think of anywhere else where you believe hooks would be useful, or if you can think of any specific requirements for your current ideas, feel free to send them over and we'll start collating everything. 

 

Thanks, 

 

Damian

 

 

 

On Wed, Sep 17, 2014 at 8:03 PM, Alpert, Reed <reed.alpert@...> wrote:

Hi,

 

I agree, the callback to a module that can do publish checking/etc is good.

 

I prototyped a checking module, and it works, although loading the dictionary on 1st publish has a small delay, and also required that the publish not be from a mama timer callback, since that deadlocked either on sending the subscription on the throttle queue, or getting the dict back.

 

We actually have more requirements for publish that are very specific to our env (adding system audit fields to all publishes to track when/where the data comes from).

Having a callback we can engage would allow us to do this w/o any other users needing to have our stuff in the way.

So loading our module at runtime and calling it would work quite well.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Thursday, September 11, 2014 5:20 AM
To: Alpert, Reed; Benjamin Taieb; Frank Quinn; fquinn.ni@...
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Hi Alpert, 

I like #2 more, but I still want to keep the more data model/market data/custom logic away from the core messaging part of OpenMAMA. 

 

What I’m thinking of is a hook/callback to a programmer defined function that would take the publisher, transport, message, a user defined closure etc that would return a status to indicate if the message should be published or not. The dictionary etc could be obtained at initialisation time through a separate init function. 

 

This dovetails with some of the other pieces of work going on:

  • we now have a framework for loading bridges dynamically, so the functionality could be in a library loaded automatically at startup
  • the hook above looks very much like publisher side entitlements (pass a publisher, message and expect a yes or no back) so we do 

 

I still need to work though this a bit more but let me know what you think.

 

Glenn 

 

From: <Alpert>, Reed <reed.alpert@...>
Date: Wednesday, 10 September 2014 19:13
To: Benjamin Taieb <benjamin.taieb@...>, Frank Quinn <fquinn@...>, "fquinn.ni@..." <fquinn.ni@...>, Glenn McClements <gmcclements@...>
Cc: "Openmama-dev@..." <Openmama-dev@...>
Subject: RE: [Openmama-dev] Enforcing field type on publish

 

Hi,

 

Yes I agree with Ben’s comments.

Here are some caveats:

1.      This is a feature that is requested by our market data infrastructure team to prevent app dev teams from accidentally publishing incorrect field types. We don’t believe that a voluntary program will work, which is why the FieldCache as a location is not preferred. Apps won’t use the FieldCache if they don’t need it, or are using an Excel plugin for publishing (quite common).

2.      This is a configured feature, and only for publication.

3.      This is not for subscription, that is, a subscriber can use any of the getXX() methods that work.

 

There seems to be general agreement that the OM layer is much better place for this feature than bridges themselves.

 

The two best locations seem to be:

1.      In mamaMsg, in the addXX and updateXX methods, probably with a macro like CHECK_MODIFY() that calls out to another module to do the work.

a.       This is good since it keep the message construction in mamaMsg module.

b.      This is good since it give the programmer the quickest feedback on an error condition.

c.       This is bad since it puts some performance hit on construction of every message, even on subscribe.

d.      This is bad since it makes the code active for creating subscribe messages too.

e.       This is bad since it puts more code in more places for this feature.

2.      In publisher.send() that checks a config var and then calls out to another module to do the work.

a.       This is good since it has minimal code changes, and minimal performance effect.

b.      This is good since it only affects publish messages.

c.       This is bad since it gives delayed failure, and there may be multiple fields that fail.

                                                              i.      Probably logging each failed field and returning an error code will work well to help developers fix problems.

 

I think the publisher.send() method is the best place, given the above.

 

But figuring out which dictionary to use for the checking seems to be the most difficult part.

Adding this feature as a config option may require also adding some optional config info about how to get a dictionary for a specific bridge, e.g., the source name.

 

Once the dictionary loading/mapping is solved then the rest seems fairly straightforward, except for the determination of allowed/denied conversions as configurable policy, but starting with 3 simple ones seems best:

1.      No checking (default)

2.      Type must match dictionary.

3.      Type must match dictionary, with a small number of allowed conversions that do not incur data loss (e.g., F32 -> F64).

 

Thanks,

 

Reed.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.