Re: Enforcing field type on publish

Alpert, Reed <reed.alpert@...>



I agree, the callback to a module that can do publish checking/etc is good.


I prototyped a checking module, and it works, although loading the dictionary on 1st publish has a small delay, and also required that the publish not be from a mama timer callback, since that deadlocked either on sending the subscription on the throttle queue, or getting the dict back.


We actually have more requirements for publish that are very specific to our env (adding system audit fields to all publishes to track when/where the data comes from).

Having a callback we can engage would allow us to do this w/o any other users needing to have our stuff in the way.

So loading our module at runtime and calling it would work quite well.






Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...


Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...


From: Glenn McClements [mailto:gmcclements@...]
Sent: Thursday, September 11, 2014 5:20 AM
To: Alpert, Reed; Benjamin Taieb; Frank Quinn;
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish


Hi Alpert, 

I like #2 more, but I still want to keep the more data model/market data/custom logic away from the core messaging part of OpenMAMA. 


What I’m thinking of is a hook/callback to a programmer defined function that would take the publisher, transport, message, a user defined closure etc that would return a status to indicate if the message should be published or not. The dictionary etc could be obtained at initialisation time through a separate init function. 


This dovetails with some of the other pieces of work going on:

  • we now have a framework for loading bridges dynamically, so the functionality could be in a library loaded automatically at startup
  • the hook above looks very much like publisher side entitlements (pass a publisher, message and expect a yes or no back) so we do 


I still need to work though this a bit more but let me know what you think.




From: <Alpert>, Reed <reed.alpert@...>
Date: Wednesday, 10 September 2014 19:13
To: Benjamin Taieb <benjamin.taieb@...>, Frank Quinn <fquinn@...>, "" <>, Glenn McClements <gmcclements@...>
Cc: "Openmama-dev@..." <Openmama-dev@...>
Subject: RE: [Openmama-dev] Enforcing field type on publish




Yes I agree with Ben’s comments.

Here are some caveats:

1.      This is a feature that is requested by our market data infrastructure team to prevent app dev teams from accidentally publishing incorrect field types. We don’t believe that a voluntary program will work, which is why the FieldCache as a location is not preferred. Apps won’t use the FieldCache if they don’t need it, or are using an Excel plugin for publishing (quite common).

2.      This is a configured feature, and only for publication.

3.      This is not for subscription, that is, a subscriber can use any of the getXX() methods that work.


There seems to be general agreement that the OM layer is much better place for this feature than bridges themselves.


The two best locations seem to be:

1.      In mamaMsg, in the addXX and updateXX methods, probably with a macro like CHECK_MODIFY() that calls out to another module to do the work.

a.       This is good since it keep the message construction in mamaMsg module.

b.      This is good since it give the programmer the quickest feedback on an error condition.

c.       This is bad since it puts some performance hit on construction of every message, even on subscribe.

d.      This is bad since it makes the code active for creating subscribe messages too.

e.       This is bad since it puts more code in more places for this feature.

2.      In publisher.send() that checks a config var and then calls out to another module to do the work.

a.       This is good since it has minimal code changes, and minimal performance effect.

b.      This is good since it only affects publish messages.

c.       This is bad since it gives delayed failure, and there may be multiple fields that fail.

                                                              i.      Probably logging each failed field and returning an error code will work well to help developers fix problems.


I think the publisher.send() method is the best place, given the above.


But figuring out which dictionary to use for the checking seems to be the most difficult part.

Adding this feature as a config option may require also adding some optional config info about how to get a dictionary for a specific bridge, e.g., the source name.


Once the dictionary loading/mapping is solved then the rest seems fairly straightforward, except for the determination of allowed/denied conversions as configurable policy, but starting with 3 simple ones seems best:

1.      No checking (default)

2.      Type must match dictionary.

3.      Type must match dictionary, with a small number of allowed conversions that do not incur data loss (e.g., F32 -> F64).





This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to for disclosures relating to European legal entities.

Join to automatically receive all group messages.