Hi Ben,

There are a few things to consider here:

1.       The MAMA Dictionary is optional too. If you look at basic mamapublisherc / mamasubscriberc, you’ll see it doesn’t use the dictionary, but it does use mamaMsg.

2.       Unit tests actually do exist for the various mandatory castings required. If you run the payload unit tests today, for example, you’ll find that getting a date time from a string as well as a date time field should be supported, as well as other payload level casting expectations… In fact if you haven’t run these tests already, you really should, especially if you want to use MAMDA. If you look at the Qpid bridge, you’ll see a bunch of code put in to make this casting more forgiving.

3.       While iterating over a message, in the interest of performance, you probably don’t want to hit the data dictionary on every field being processed, particularly if you’re trying to do high performance field caching.

Basically, I’m of the opinion that if you want to make a payload flexible with respect to the field types it supports, that should be done at a payload layer and we have unit tests to verify the mandatory minimum.

If you want to enforce MAMA Dictionary compliance, this should be done in the MAMA application, where you can use field descriptors to verify content type at send and receive sides. E.g.

switch (mamaFieldDescriptor_getType (fieldDescriptor))

{

    case MAMA_FIELD_TYPE_I32:

    case MAMA_FIELD_TYPE_I64:

        allow the add / access;

    default:

        handle condition;

}

As well as everything else, whether or not the I64 to I32 casting (yes - the other way around) will be legal will depend on the data content being cast and sent across. Depending on the content, a U8 / U16 could even be fine. Without knowledge and consideration of this within the application, undesired truncation of data could be found, or opportunities to avoid failed data transfer due to strictness checking could be avoided.

Cheers,

Frank

Just to put emphasis on what Reed explain:

It should be probably implemented in the C layer, making sure that it "catches" all languages "for free".

The goal is to enforce it in OpenMama layer, so that OpenMama application are forced to adheres to it.

I believe use of MamaFieldCache is optional, which would rules it out.

And referring to Glenn comments about mamaMsg, the goal is that the payload does not trust the calling application.

There is multiples challenges I believe in implementing that:

-How does the mama layer get dictionary ? This could be easily solves by using dictionnary requestor wherever we decides to implement it.

-How does the mapping between a particular msg and namespace is going to work ? At the moment, a dictionary is attached to namespace, namespace could be shared across multiples sources, but a message is source agnostic, only the publisher when it send the msg will send on a particular source/ symbol. On the other side, we would want to fail as soon as possible in the process, probably in the mamaMsg_addXX and mamaMsg_setXX calls, so that the calling application know for which fields it is doing something wrong.

-We need to clarify the "authorised cast": If you're doing addI32 on a I64 fields, should probably get authorised. At the moment, various Payload implement various level of flexibility, would be good that we defines UnitTests defining precisely what it should be, probably modelled from what is defined in wombatmsg, for retro-compatibility purpose.

I'll take any input from the community, particularly on point 2, which is the hardest in my opinion.

Cheers,

Ben.

Yes, this is C++, but we also have Java and DotNet clients (some C# and some Excel).

We do publish best practices, but our app dev groups sometimes do not follow them, and we want to prevent the SolCache from getting out of sync with the dictionary.

We also provide them with sample apps that have this type of control, but sometimes … J

Thanks,

Reed.

Reed Alpert | Corporate & Investment Bank | Market Data Services | J.P. Morgan | 4 Metrotech Center, 23rd Floor, Brooklyn, NY 11245 | T: 718.242.5198  | M: 917.414.4613 | reed.alpert@...

Alternate Contact:  CIB PIM Trading Technology Solutions NA | CIB_PIM_Trading_Technology_Solutions_NA@...

Hi,

Attached is a patch for OpenMAMA JNI that overloads the MamaPublisher create method. For consistency purposes, the new method requires two String parameters, source and symbol, as opposed to a single string currently found in the MamaPublisher class.  

diff --git a/mama/jni/src/c/mamapublisherjni.c b/mama/jni/src/c/mamapublisherjni.c

index 518bacf..629a51b 100644

--- a/mama/jni/src/c/mamapublisherjni.c

+++ b/mama/jni/src/c/mamapublisherjni.c

@@ -78,7 +78,7 @@ static void MAMACALLTYPE sendCompleteCB (mamaPublisher publisher,

  * Method:    _create

  * Signature: (Lcom/wombat/mama/Transport;Ljava/lang/String;)V

  */

-JNIEXPORT void JNICALL Java_com_wombat_mama_MamaPublisher__1create

+JNIEXPORT void JNICALL Java_com_wombat_mama_MamaPublisher__1create__Lcom_wombat_mama_MamaTransport_2Ljava_lang_String_2

   (JNIEnv* env, jobject this, jobject transport, jstring topic)

{

     mamaPublisher   cPublisher          =   NULL;

@@ -125,6 +125,66 @@ JNIEXPORT void JNICALL Java_com_wombat_mama_MamaPublisher__1create

     return;

}

+ /*

+ * Class:     com_wombat_mama_MamaPublisher

+ * Method:    _create

+ * Signature: (Lcom/wombat/mama/MamaTransport;Ljava/lang/String;Ljava/lang/String;)V

+ */

+JNIEXPORT void JNICALL Java_com_wombat_mama_MamaPublisher__1create__Lcom_wombat_mama_MamaTransport_2Ljava_lang_String_2Ljava_lang_String_2

+  (JNIEnv* env, jobject this, jobject transport, jstring source, jstring symbol)

+{

+             mamaPublisher   cPublisher          =   NULL;

+    mamaTransport   cTransport          =   NULL;

+    const char*     cSource             =   NULL;

+             const char*     cSymbol             =   NULL;

+    jlong           transportPointer    =   0;

+    mama_status     status              =   MAMA_STATUS_OK;

+    char errorString[UTILS_MAX_ERROR_STRING_LENGTH];

+            

+    /*Get the transport pointer*/

+    assert(transport!=NULL);

+    transportPointer = (*env)->GetLongField(env, transport,

+            transportPointerFieldId_g);

+    cTransport = CAST_JLONG_TO_POINTER(mamaTransport, transportPointer);

+    assert(transportPointer!=0);

+            

+    /*Get the char* from the jstring*/

+    if(NULL!=source)

+    {

+         cSource = (*env)->GetStringUTFChars(env,source,0);

+         if(!cSource)return;

+    }

+             if(NULL!=symbol)

+    {

+         cSymbol = (*env)->GetStringUTFChars(env,symbol,0);

+         if(!cSymbol)return;

+    }

+

+    if(MAMA_STATUS_OK!=(mamaPublisher_create(

+                    &cPublisher,

+                    cTransport,

+                    cSymbol,

+                    cSource,

+                    NULL)))

+    {

+        if(cSource)(*env)->ReleaseStringUTFChars(env,source, cSource);

+                             if(cSymbol)(*env)->ReleaseStringUTFChars(env,symbol, cSymbol);

+        utils_buildErrorStringForStatus(

+                errorString, UTILS_MAX_ERROR_STRING_LENGTH,

+                "Failed to create publisher.", status);

+        utils_throwMamaException(env,errorString);

+        return;

+    }

+

+    (*env)->SetLongField(env,this,publisherPointerFieldId_g,

+                         CAST_POINTER_TO_JLONG(cPublisher));

+       

+    if(cSource)(*env)->ReleaseStringUTFChars(env,source, cSource);

+             if(cSymbol)(*env)->ReleaseStringUTFChars(env,symbol, cSymbol);

+   

+    return;

+}

+

/*

  * Class:     com_wombat_mama_MamaPublisher

  * Method:    _send

diff --git a/mama/jni/src/com/wombat/mama/MamaPublisher.java b/mama/jni/src/com/wombat/mama/MamaPublisher.java

index 0146945..c1e8dc8 100644

--- a/mama/jni/src/com/wombat/mama/MamaPublisher.java

+++ b/mama/jni/src/com/wombat/mama/MamaPublisher.java

@@ -39,6 +39,11 @@ public class MamaPublisher

     {

         _create(transport,topic);

     }

+            

+             public void create (MamaTransport transport, String source, String symbol)

+    {

+        _create(transport,source,symbol);

+    }

     public void send (MamaMsg msg)

     {

@@ -77,6 +82,8 @@ public class MamaPublisher

     }

     private native void _create (MamaTransport transport, String topic);

+            

+             private native void _create (MamaTransport transport, String source, String symbol);

     private native void _send (MamaMsg msg);

--

1.8.4.msysgit.0

Chad Meyer | Corporate & Investment Bank | PIM Market Data Services | J.P. Morgan | 4 Metrotech Center 23rd Floor Brooklyn, NY 11201 | Tel: (718) 242-5165 | M: (215) 801-2606 | chad.j.meyer@...

Hi folks,

I noticed that VectorChar and VectorBool do not have  a field type enumerator  in Mama (i.e. mamaFieldType enum in mama/fielddesc.h). Also, the related code for VectorChar and VectorBool in C/ C++, C#, JNI does not have the type related implementations for these vector types as compared to other vector types, such as VectorU32 (enumerator: MAMA_FIELD_TYPE_VECTOR_U32).

I imagine these types may not be widely and there are alternatives field types for them(e.g. Opaque, String and unsinged integer).

Should VectorChar and VectorBool  types be included in the mamaFieldType enum? For example with the numbers shown below:

<code>

diff --git a/mama/c_cpp/src/c/mama/fielddesc.h b/mama/c_cpp/src/c/mama/fielddesc.h

index e3af6e1..a374489 100644

--- a/mama/c_cpp/src/c/mama/fielddesc.h

+++ b/mama/c_cpp/src/c/mama/fielddesc.h

@@ -90,6 +90,8 @@ typedef enum mamaFieldType_

     MAMA_FIELD_TYPE_PRICE         =   27,

     /** Array type support */

+    MAMA_FIELD_TYPE_VECTOR_BOOL   =   29,

+    MAMA_FIELD_TYPE_VECTOR_CHAR   =   30,

     MAMA_FIELD_TYPE_VECTOR_I8     =   34,

     MAMA_FIELD_TYPE_VECTOR_U8     =   35,

     MAMA_FIELD_TYPE_VECTOR_I16    =   36,

</code>

And then also have the corresponding type related changes in the rest of Mama codebase?

Regards,.

--Alireza

Hi Glenn

I’m resubmitting that patch (attached and below). This now conforms with your comments

Tom

diff --git a/README b/README

index be1ad38..f278d0d 100644

--- a/README

+++ b/README

@@ -2,7 +2,7 @@ OpenMAMA Client Library

============================

 Release: 2.3.1

-Change Log: http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/OpenMAMA-2.3.1

+Change Log: http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/tags/OpenMAMA-2.3.1-release

 Project Page

------------

diff --git a/mama/c_cpp/src/c/dictionary.c b/mama/c_cpp/src/c/dictionary.c

index 7f7d605..a4e6cdf 100644

--- a/mama/c_cpp/src/c/dictionary.c

+++ b/mama/c_cpp/src/c/dictionary.c

@@ -495,6 +495,20 @@ mamaDictionary_getDictionaryMessage (

     return MAMA_STATUS_OK;

}

+mama_status mamaDictionary_populateDictionaryMessage(

+             mamaDictionary dictionary,

+             mamaMsg msg)

+{

+             mamaDictionaryImpl* impl = (mamaDictionaryImpl*)dictionary;

+

+             if (!impl) return MAMA_STATUS_NULL_ARG;

+             if (!msg) return MAMA_STATUS_INVALID_ARG;

+

+             populateMessageFromDictionary (impl, msg);

+

+             return MAMA_STATUS_OK;

+}

+

mama_status

mamaDictionary_createFieldDescriptor (

                 mamaDictionary       dictionary,

diff --git a/mama/c_cpp/src/c/mama/dictionary.h b/mama/c_cpp/src/c/mama/dictionary.h

index 0eda958..faa7fe8 100644

--- a/mama/c_cpp/src/c/mama/dictionary.h

+++ b/mama/c_cpp/src/c/mama/dictionary.h

@@ -268,6 +268,20 @@ mamaDictionary_getDictionaryMessage (

     mamaMsg* msg);

 /**

+ * Fill a message with the data dictionary.

+ *

+ * An existing mama message is populated fromt he data dictionary

+ * This allows the caller to create the message on a specifc payload

+ *

+ * @param dictionary The Dictionary

+ * @param msg The address of the mamaMsg where the result is to be written

+ */

+MAMAExpDLL extern mama_status mamaDictionary_populateDictionaryMessage (

+    mamaDictionary dictionary,

+    mamaMsg msg);

+

+

+/**

  * Create a new field descriptor and add it to the dictionary.

  * New fields can be added to an existing dictionary obtained

  * from the MAMA infrastructure. This function can also be used

Hi,

Yes, the addString() is from MamaMsg.

The cache is our own market data infrastructure, not the MamaFieldCache.

In this case it is either Solace/SolCache or TREP RTIC.

We want business app dev groups to be able to look at the dictionary, see a field that is MamaPrice, and be guaranteed that MamaMsg.getPrice() will always work (although MamaPrice.getIsValidPrice() may return false).

A certain number of the type mismatches come from Excel, where a failed formula may populate a cell as #NA, and that gets published into a price/float field.

Thanks,

Reed.

Hi,

This is regarding a way to check that the field data type being published matches (closely) the dictionary data type.

The motivation for this is to prevent our internally published data caches from invalid data.

For example, if a field (name=MARK_PRICE fid=300) is a MamaPrice in the dictionary, and an app uses this:

                addString("MARK_PRICE", 300, "invalid price");

then we have a field that is now a string in the cache, and that causes problems with subscribing apps that use getPrice().

Currently we are looking to the bridges that we use (Tick42/TREP & Solace) to enforce these restrictions.

Adding this to the OpenMAMA layer is tricky, but allows a single policy to be enforced across multiple bridges. The main question from my point of view being how to set the policy. They can range from none (default), to most strict (published field's type must match dictionary), to common conversions supported (e.g., F32 -> F64), to a custom grid setting exactly what conversions are allowed.

I welcome any input from others who have run into this same issue, or found other ways to solve it.

Thanks,

Reed.

Hi Glenn

Both of those of those points are accepted and I’ll re-submit a patch in a couple of days when I can get this back to the top of my todo list J

Is this function ever used anywhere other than in a bridge? We only use it in the context of a bridge building a message from the dictionary in order to publish it

Tom

Hi

Attached is a patch for OpenMAMA that provides an alternative function to populate a mamaMsg with the dictionary. This differs from the existining function by allowing the caller to pass a mamaMsg that has been pre-created on whatever payload is desired.

We are currently building and running the Tick42 RMDS bridge against OpenMAMA libraries with the patch applied. The new function is used something like this (with conditional compilation to show the ‘old’ version)

              // Prepare the OpenMAMA dictionary part of the message

#if defined(MAMA_DICTIONARY_PATCH)

              // provided mama has been patched for the new dictionary function we can create our own message on our own payload

              mamaMsg dictMsg=NULL;

              mamaMsg_createForPayload(&dictMsg, 'P');

              mama_status res = mamaDictionary_fillDictionaryMessage(dictionary, &dictMsg);

#else

              mamaMsg dictMsg=NULL;

              mama_status res = mamaDictionary_getDictionaryMessage (dictionary, &dictMsg);

#endif

As I said before, in theory, it shouldn’t matter which payload a dictionary message uses, but in a context when multiple bridges are loaded it’s desirable to know which code is being executed.

I’m interested to hear from other bridge developers any thoughts, opiniions or questions on this subject

Best regards

Tom

The patch is also shown below

diff --git a/README b/README

index be1ad38..f278d0d 100644

--- a/README

+++ b/README

@@ -2,7 +2,7 @@ OpenMAMA Client Library

============================

 Release: 2.3.1

-Change Log: http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/heads/OpenMAMA-2.3.1

+Change Log: http://git.openmama.org/?p=OpenMAMA.git;a=shortlog;h=refs/tags/OpenMAMA-2.3.1-release

 Project Page

------------

diff --git a/mama/c_cpp/src/c/dictionary.c b/mama/c_cpp/src/c/dictionary.c

index 7f7d605..8a6d275 100644

--- a/mama/c_cpp/src/c/dictionary.c

+++ b/mama/c_cpp/src/c/dictionary.c

@@ -496,6 +496,21 @@ mamaDictionary_getDictionaryMessage (

}

 mama_status

+             mamaDictionary_fillDictionaryMessage (

+             mamaDictionary dictionary,

+             mamaMsg* msg)

+{

+             mamaDictionaryImpl* impl = (mamaDictionaryImpl*)dictionary;

+

+             if (!impl) return MAMA_STATUS_NULL_ARG;

+             if (!msg) return MAMA_STATUS_INVALID_ARG;

+

+             populateMessageFromDictionary (impl, *msg);

+

+             return MAMA_STATUS_OK;

+}

+

+mama_status

mamaDictionary_createFieldDescriptor (

                 mamaDictionary       dictionary,

                 mama_fid_t           fid,

diff --git a/mama/c_cpp/src/c/mama/dictionary.h b/mama/c_cpp/src/c/mama/dictionary.h

index 0eda958..40f7885 100644

--- a/mama/c_cpp/src/c/mama/dictionary.h

+++ b/mama/c_cpp/src/c/mama/dictionary.h

@@ -268,6 +268,22 @@ mamaDictionary_getDictionaryMessage (

     mamaMsg* msg);

 /**

+ * Fill a message with the data dictionary.

+ *

+ * An existing mama message is populated fromt he data dictionary

+ * This allows the caller to create the message on a specifc payload

+ *

+ * @param dictionary The Dictionary

+ * @param msg The address of the mamaMsg where the result is to be written

+ */

+MAMAExpDLL

+extern mama_status

+mamaDictionary_fillDictionaryMessage (

+    mamaDictionary dictionary,

+    mamaMsg* msg);

+

+

+/**

  * Create a new field descriptor and add it to the dictionary.

  * New fields can be added to an existing dictionary obtained

  * from the MAMA infrastructure. This function can also be used