Re: Enforcing field type on publish


Alpert, Reed <reed.alpert@...>
 

Hi Damian,

 

I have our pre-publish check code ready, but it is not pluggable.

I’d like to submit it as a patch using #ifdef like the entitlements code.

That way I don’t have to maintain it as OM evolves, and it won’t affect other OM developers.

When the pluggable/hook system is ready I can port it over.

 

There are 2 new files:

mama/c_cpp/src/c/publisherfieldcheck.c

mama/c_cpp/src/c/publisherfieldcheck.h

 

and changes to:

mama/c_cpp/src/c/Sconscript

mama/c_cpp/src/c/transport.c

mama/c_cpp/src/c/mama/transport.h

mama/c_cpp/src/c/publisher.c

 

Please let me know if an #ifdef patch is OK for this.

I can submit the patch if you want to see what it involves.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Alpert, Reed
Sent: Monday, September 22, 2014 6:17 PM
To: 'Damian Maguire'
Cc: Glenn McClements; Benjamin Taieb; Frank Quinn; fquinn.ni@...; Openmama-dev@...
Subject: RE: [Openmama-dev] Enforcing field type on publish

 

Hi Damian,

 

Yes, having a pluggable module that handles some pre/post events makes sense.

Our handler for this now is very JPM-specific, and I don’t think it would make sense to try and standardize that (checks field types, ensures there is a msgType, and adds some audit fields to track who/when/where published the msg).

 

To get a dictionary I put a hook in at the end of the transport create, since that is the first time we have enough info to check the config in mama.properties and then subscribe to the dictionary.

If that is OK then the dictionary is stashed in the transport data, and the pre-publish looks at that to see if it should apply the checks.

 

Begin able to load a shared obj via the library manager would work great.

For the pre/post events it seems like a lot to have a general list like transport create and publish and maybe bridge create, etc.

I’m not sure if that is overkill to fully fill out the list of pre/post events across many objects.

 

To store library data in the main objects’ data (e.g., bridge, transport, publisher, etc) maybe a user closure can be added.

Of course that assumes that a single pre/post library is installed.

 

I’ll keep adding things as our development continue.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Damian Maguire [mailto:damian@...]
Sent: Friday, September 19, 2014 11:53 AM
To: Alpert, Reed
Cc: Glenn McClements; Benjamin Taieb; Frank Quinn; fquinn.ni@...; Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Hey Reed, 

 

This all sounds really interesting, particularly the fact that you have already identified multiple use cases for publish side hooks. A lot of it actually ties very nicely with some of the internal discussions we've been having around a plugin architecture for OpenMAMA as well, which in turn sit quite nicely with the library manager work which is also ongoing. 

 

At present myself and the team here are pretty solidly booked with a few internal projects (which should provide some real benefits to OpenMAMA when they're complete), so it'll be a few weeks before we can sit down and tackle this properly. However, I'd be very keen to get some full community engagement in discussing this, and work to put together a design that covers the requirements of the majority of potential use cases. With that in mind, if you guys can think of anywhere else where you believe hooks would be useful, or if you can think of any specific requirements for your current ideas, feel free to send them over and we'll start collating everything. 

 

Thanks, 

 

Damian

 

 

 

On Wed, Sep 17, 2014 at 8:03 PM, Alpert, Reed <reed.alpert@...> wrote:

Hi,

 

I agree, the callback to a module that can do publish checking/etc is good.

 

I prototyped a checking module, and it works, although loading the dictionary on 1st publish has a small delay, and also required that the publish not be from a mama timer callback, since that deadlocked either on sending the subscription on the throttle queue, or getting the dict back.

 

We actually have more requirements for publish that are very specific to our env (adding system audit fields to all publishes to track when/where the data comes from).

Having a callback we can engage would allow us to do this w/o any other users needing to have our stuff in the way.

So loading our module at runtime and calling it would work quite well.

 

Thanks,

 

Reed.

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

 

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Thursday, September 11, 2014 5:20 AM
To: Alpert, Reed; Benjamin Taieb; Frank Quinn; fquinn.ni@...
Cc: Openmama-dev@...
Subject: Re: [Openmama-dev] Enforcing field type on publish

 

Hi Alpert, 

I like #2 more, but I still want to keep the more data model/market data/custom logic away from the core messaging part of OpenMAMA. 

 

What I’m thinking of is a hook/callback to a programmer defined function that would take the publisher, transport, message, a user defined closure etc that would return a status to indicate if the message should be published or not. The dictionary etc could be obtained at initialisation time through a separate init function. 

 

This dovetails with some of the other pieces of work going on:

  • we now have a framework for loading bridges dynamically, so the functionality could be in a library loaded automatically at startup
  • the hook above looks very much like publisher side entitlements (pass a publisher, message and expect a yes or no back) so we do 

 

I still need to work though this a bit more but let me know what you think.

 

Glenn 

 

From: <Alpert>, Reed <reed.alpert@...>
Date: Wednesday, 10 September 2014 19:13
To: Benjamin Taieb <benjamin.taieb@...>, Frank Quinn <fquinn@...>, "fquinn.ni@..." <fquinn.ni@...>, Glenn McClements <gmcclements@...>
Cc: "Openmama-dev@..." <Openmama-dev@...>
Subject: RE: [Openmama-dev] Enforcing field type on publish

 

Hi,

 

Yes I agree with Ben’s comments.

Here are some caveats:

1.      This is a feature that is requested by our market data infrastructure team to prevent app dev teams from accidentally publishing incorrect field types. We don’t believe that a voluntary program will work, which is why the FieldCache as a location is not preferred. Apps won’t use the FieldCache if they don’t need it, or are using an Excel plugin for publishing (quite common).

2.      This is a configured feature, and only for publication.

3.      This is not for subscription, that is, a subscriber can use any of the getXX() methods that work.

 

There seems to be general agreement that the OM layer is much better place for this feature than bridges themselves.

 

The two best locations seem to be:

1.      In mamaMsg, in the addXX and updateXX methods, probably with a macro like CHECK_MODIFY() that calls out to another module to do the work.

a.       This is good since it keep the message construction in mamaMsg module.

b.      This is good since it give the programmer the quickest feedback on an error condition.

c.       This is bad since it puts some performance hit on construction of every message, even on subscribe.

d.      This is bad since it makes the code active for creating subscribe messages too.

e.       This is bad since it puts more code in more places for this feature.

2.      In publisher.send() that checks a config var and then calls out to another module to do the work.

a.       This is good since it has minimal code changes, and minimal performance effect.

b.      This is good since it only affects publish messages.

c.       This is bad since it gives delayed failure, and there may be multiple fields that fail.

                                                              i.      Probably logging each failed field and returning an error code will work well to help developers fix problems.

 

I think the publisher.send() method is the best place, given the above.

 

But figuring out which dictionary to use for the checking seems to be the most difficult part.

Adding this feature as a config option may require also adding some optional config info about how to get a dictionary for a specific bridge, e.g., the source name.

 

Once the dictionary loading/mapping is solved then the rest seems fairly straightforward, except for the determination of allowed/denied conversions as configurable policy, but starting with 3 simple ones seems best:

1.      No checking (default)

2.      Type must match dictionary.

3.      Type must match dictionary, with a small number of allowed conversions that do not incur data loss (e.g., F32 -> F64).

 

Thanks,

 

Reed.


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.


_______________________________________________
Openmama-dev mailing list
Openmama-dev@...
https://lists.openmama.org/mailman/listinfo/openmama-dev

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.

Join Openmama-dev@lists.openmama.org to automatically receive all group messages.