OpenMAMA Data Quality as pluggable libs


Alpert, Reed <reed.alpert@...>
 

Hi,

 

I've been looking at the data quality (dq) code and have some ideas on how to support multiple dq strategies.

 

Generally mama has a dq strategy built into dqstrategy.c, subscription.c, and transport.c that supports:

·         SeqNums being present from the publisher to detect gaps and request recaps.

·         SenderIds being present from the publisher and used for FT takeover.

·         Initial/Recap pre-cache to correctly order an initial/recap and updates.

·         Various config parameters for how the app should handle dq issues:

o   Transport bounces and setting subs stale.

o   Duplicate msgs.

o   When to apply pre-cache.

o   Handle initials (requiresInitial).

o   Turn on/off dq (recoverGaps).

 

In order to provide additional dq strategies it is proposed:

·         Move any code that references dq variables into dqstrategy.c. This is largely done, but there is some code in subscription.c that can be moved.

·         Make all dq structures opaque so they are only visible via methods.

·         Turn dqstrategy.c into a pluggable dq bridge, using entitlements as a model.

At this point current impl will be the default pluggable dq bridge, so the same behavior as now will be observed, and additional pluggable dq libs can be developed.

 

There is also the code in transport.c that handles transport bounces. This needs to be defined as to what service it provides for the bridges.

·         When a transport disconnects it can notify the app via a transport callback, and also each sub on the transport via stale/maybeStale.

·         When a transport reconnects it can notify the app via a transport callback, and also each sub via quality=OK, and requesting a recap.

It may be helpful to create an mDQTransport struct to hold transport-specific dq vars.

 

Attached is the beginnings of a state diagram for dq - it is a bit busy, but I think will be helpful in determining what does happen. Note that this does have paths for the code we added to send stale to subs even if dq is turned off. The red text/lines are for the 99.9% use case.

 

Any comments or ways forward please contribute.

 

Thanks,

 

Reed.

 

 


Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

Alternate Contact:  CIB Market Data Services Engineers | CIB_Market_Data_Services_Engineers@...

 

 

This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.

Join Openmama-dev@lists.openmama.org to automatically receive all group messages.