Re: Enforcing field type on publish


Glenn McClements <gmcclements@...>
 

Hi Alpert, 
I like #2 more, but I still want to keep the more data model/market data/custom logic away from the core messaging part of OpenMAMA. 

What I’m thinking of is a hook/callback to a programmer defined function that would take the publisher, transport, message, a user defined closure etc that would return a status to indicate if the message should be published or not. The dictionary etc could be obtained at initialisation time through a separate init function. 

This dovetails with some of the other pieces of work going on:
  • we now have a framework for loading bridges dynamically, so the functionality could be in a library loaded automatically at startup
  • the hook above looks very much like publisher side entitlements (pass a publisher, message and expect a yes or no back) so we do 

I still need to work though this a bit more but let me know what you think.

Glenn 

From: <Alpert>, Reed <reed.alpert@...>
Date: Wednesday, 10 September 2014 19:13
To: Benjamin Taieb <benjamin.taieb@...>, Frank Quinn <fquinn@...>, "fquinn.ni@..." <fquinn.ni@...>, Glenn McClements <gmcclements@...>
Cc: "Openmama-dev@..." <Openmama-dev@...>
Subject: RE: [Openmama-dev] Enforcing field type on publish

Hi,

 

Yes I agree with Ben’s comments.

Here are some caveats:

1.       This is a feature that is requested by our market data infrastructure team to prevent app dev teams from accidentally publishing incorrect field types. We don’t believe that a voluntary program will work, which is why the FieldCache as a location is not preferred. Apps won’t use the FieldCache if they don’t need it, or are using an Excel plugin for publishing (quite common).

2.       This is a configured feature, and only for publication.

3.       This is not for subscription, that is, a subscriber can use any of the getXX() methods that work.

 

There seems to be general agreement that the OM layer is much better place for this feature than bridges themselves.

 

The two best locations seem to be:

1.       In mamaMsg, in the addXX and updateXX methods, probably with a macro like CHECK_MODIFY() that calls out to another module to do the work.

a.       This is good since it keep the message construction in mamaMsg module.

b.      This is good since it give the programmer the quickest feedback on an error condition.

c.       This is bad since it puts some performance hit on construction of every message, even on subscribe.

d.      This is bad since it makes the code active for creating subscribe messages too.

e.      This is bad since it puts more code in more places for this feature.

2.       In publisher.send() that checks a config var and then calls out to another module to do the work.

a.       This is good since it has minimal code changes, and minimal performance effect.

b.      This is good since it only affects publish messages.

c.       This is bad since it gives delayed failure, and there may be multiple fields that fail.

                                                               i.      Probably logging each failed field and returning an error code will work well to help developers fix problems.

 

I think the publisher.send() method is the best place, given the above.

 

But figuring out which dictionary to use for the checking seems to be the most difficult part.

Adding this feature as a config option may require also adding some optional config info about how to get a dictionary for a specific bridge, e.g., the source name.

 

Once the dictionary loading/mapping is solved then the rest seems fairly straightforward, except for the determination of allowed/denied conversions as configurable policy, but starting with 3 simple ones seems best:

1.       No checking (default)

2.       Type must match dictionary.

3.       Type must match dictionary, with a small number of allowed conversions that do not incur data loss (e.g., F32 -> F64).

 

Thanks,

 

Reed.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

Join Openmama-dev@lists.openmama.org to automatically receive all group messages.