[PATCH] MAMA: Add the capability to load payload libraries just-in-time (JIT).


Lee Skillen <lskillen@...>
 

If a library hasn't been previously loaded then OpenMAMA will now attempt
to load it by converting the payload identifier into a well-known string
for the payload library. This behaviour is useful if the payload isn't
known ahead of time or if lazy loading is desired over enumeration of a
existing set of libraries. No changes made to existing API interfaces.

The following functions have been added :-

- mamaPayload_convertToLibString():
Maps a mamaPayloadType enumeration to the well-known library name.

- mamaPayload_convertFromLibString():
Inverse of mamaPayload_convertToLibString.

- mama_loadPayloadBridgeByType():
Takes a mamaPayloadType and utilises mamaPayload_convertToLibString()
to retrieve the library name for the payload, and then delegates this
to the existing mama_loadPayloadBridge() function.

The string conversions are implemented as a linear search, primarily due
to these functions not being on the hot path - Their use is prior to a
library being loaded and (I would imagine) would be seldom used in other
situations.

Additional changes :-

If msgPayload_destroyImpl has been exposed by the payload library then
this is now utilised to cleanup the payload upon shutdown properly. It
was observed that each payload always leaked memory due to lacking the
capability to destroy itself. Individual payloads will still need to
expose this - Backwards compatible as it won't be called otherwise.

The MamaEnumTestsC set of unit tests have been rewritten to utilise the
new conversion functionality. The tests have also been renamed as such:

- testPayloadToString -> testPayloadStringConversions
- testPayloadFromString -> testPayloadStringConversions
- testMiddlewareToString -> testMiddlewareStringConversions
- testMiddlewareFromString -> testMiddlewareStringConversions

Noticed that the ASSERT_EQ/EXPECT_EQ statements in gtest are sometimes
used incorrectly, in that the expected/actual are reversed (it should be
expected on the left and actual on the right).

Todo :-

Need to confirm well-known library names with all vendors - Will only be
important if JIT loading is actually used, but the names should be made
accurate anyway.

Signed-off-by: Lee Skillen <lskillen@...>
---
mama/c_cpp/src/c/mama.c | 77 ++++++++--
mama/c_cpp/src/c/mama/mama.h | 27 +++-
mama/c_cpp/src/c/mama/msg.h | 30 ++++
mama/c_cpp/src/c/msg.c | 163 +++++++++++++++------
mama/c_cpp/src/c/payloadbridge.h | 4 +
.../src/gunittest/c/payloadmiddlewareidtest.cpp | 152 ++++++++-----------
6 files changed, 304 insertions(+), 149 deletions(-)

diff --git a/mama/c_cpp/src/c/mama.c b/mama/c_cpp/src/c/mama.c
index 33a0768..3a7d7be 100644
--- a/mama/c_cpp/src/c/mama.c
+++ b/mama/c_cpp/src/c/mama.c
@@ -28,6 +28,7 @@
#include "wombat/strutils.h"

#include <mama/mama.h>
+#include <mama/msg.h>
#include <mama/error.h>
#include <mamainternal.h>
#include <mama/version.h>
@@ -168,6 +169,7 @@ typedef struct mamaImpl_
mamaPayloadBridge myPayloads[MAMA_PAYLOAD_MAX];
LIB_HANDLE myBridgeLibraries[MAMA_MIDDLEWARE_MAX];
LIB_HANDLE myPayloadLibraries[MAMA_PAYLOAD_MAX];
+ msgPayload_destroyImpl myPayloadDestroyFuncs[MAMA_PAYLOAD_MAX];
unsigned int myRefCount;
wthread_static_mutex_t myLock;
} mamaImpl;
@@ -175,7 +177,7 @@ typedef struct mamaImpl_
static mamaApplicationContext appContext;
static char mama_ver_string[256];

-static mamaImpl gImpl = {{0}, {0}, {0}, {0}, 0, WSTATIC_MUTEX_INITIALIZER};
+static mamaImpl gImpl = {{0}, {0}, {0}, {0}, {0}, 0, WSTATIC_MUTEX_INITIALIZER};

/* ************************************************************************* */
/* Private Function Prototypes. */
@@ -624,10 +626,21 @@ mamaInternal_findBridge ()
mamaPayloadBridge
mamaInternal_findPayload (char id)
{
+ mamaPayloadBridge bridge = NULL;
+
if ('\0' == id)
return NULL;

- return gImpl.myPayloads[(uint8_t)id];
+ bridge = gImpl.myPayloads[(uint8_t)id];
+
+ if (NULL == bridge)
+ {
+ /* Dynamically load the payload bridge just-in-time, if it hasn't
+ * already been loaded previously. */
+ mama_loadPayloadBridgeByType (&bridge, (mamaPayloadType)id);
+ }
+
+ return bridge;
}

mamaPayloadBridge
@@ -1241,17 +1254,33 @@ mama_closeCount (unsigned int* count)

cleanupReservedFields();

- /* Look for a bridge for each of the payloads and close them */
+ /* Look for a bridge for each of the payloads and close them */
for (payload = 0; payload != MAMA_PAYLOAD_MAX; ++payload)
{
- /* mamaPayloadBridgeImpl* impl = (mamaPayloadBridgeImpl*)
- * gImpl.myPayloads [(uint8_t)payload];*/
- gImpl.myPayloads[(uint8_t)payload] = NULL;
- if(gImpl.myPayloadLibraries[(uint8_t)payload])
+ uint8_t id = (uint8_t)payload;
+ LIB_HANDLE lib = gImpl.myPayloadLibraries[id];
+ mamaPayloadBridgeImpl* impl = gImpl.myPayloads[id];
+
+ if (!lib)
{
- closeSharedLib (gImpl.myPayloadLibraries[(uint8_t)payload]);
- gImpl.myPayloadLibraries[(uint8_t)payload] = NULL;
+ continue;
+ }
+
+ if (impl)
+ {
+ msgPayload_destroyImpl destroyFunc =
+ gImpl.myPayloadDestroyFuncs[id];
+
+ if (destroyFunc)
+ {
+ destroyFunc (impl);
+ }
+
+ gImpl.myPayloads[id] = NULL;
}
+
+ closeSharedLib (lib);
+ gImpl.myPayloadLibraries[id] = NULL;
}

gDefaultPayload = NULL;
@@ -1827,18 +1856,29 @@ mama_setDefaultPayload (char id)

mama_status
mama_loadPayloadBridge (mamaPayloadBridge* impl,
- const char* payloadName)
+ const char* payloadName)
{
return mama_loadPayloadBridgeInternal (impl, payloadName);
}
+
+mama_status
+mama_loadPayloadBridgeByType (mamaPayloadBridge* impl,
+ mamaPayloadType payload)
+{
+ const char* name = mamaPayload_convertToLibString (payload);
+ return mama_loadPayloadBridgeInternal (impl, name);
+}
+
mama_status
mama_loadPayloadBridgeInternal (mamaPayloadBridge* impl,
const char* payloadName)
{
char bridgeImplName [256];
char initFuncName [256];
+ char destroyFuncName [256];
LIB_HANDLE bridgeLib = NULL;
msgPayload_createImpl initFunc = NULL;
+ msgPayload_destroyImpl destroyFunc = NULL;
mama_status status = MAMA_STATUS_OK;
char payloadChar ='\0';
void* vp = NULL;
@@ -1865,11 +1905,10 @@ mama_loadPayloadBridgeInternal (mamaPayloadBridge* impl,
return MAMA_STATUS_NO_BRIDGE_IMPL;
}

- snprintf (initFuncName, 256, "%sPayload_createImpl", payloadName);
-
/* Gives a warning - casting from void* to bridge_createImpl func */
- vp = loadLibFunc (bridgeLib, initFuncName);
- initFunc = *(msgPayload_createImpl*) &vp;
+ snprintf (initFuncName, 256, "%sPayload_createImpl", payloadName);
+ vp = loadLibFunc (bridgeLib, initFuncName);
+ initFunc = *(msgPayload_createImpl*) &vp;

if (!initFunc)
{
@@ -1887,7 +1926,7 @@ mama_loadPayloadBridgeInternal (mamaPayloadBridge* impl,

if (MAMA_STATUS_OK != (status = initFunc (impl, &payloadChar)))
{
- wthread_static_mutex_unlock (&gImpl.myLock);
+ wthread_static_mutex_unlock (&gImpl.myLock);

return status;
}
@@ -1910,13 +1949,19 @@ mama_loadPayloadBridgeInternal (mamaPayloadBridge* impl,
"Payload bridge %s already loaded",
payloadName);

- wthread_static_mutex_unlock (&gImpl.myLock);
+ wthread_static_mutex_unlock (&gImpl.myLock);

return MAMA_STATUS_OK;
}

+ /* Gives a warning - casting from void* to bridge_destroyImpl func */
+ snprintf (destroyFuncName, 256, "%sPayload_destroyImpl", payloadName);
+ vp = loadLibFunc (bridgeLib, destroyFuncName);
+ destroyFunc = *(msgPayload_destroyImpl*) &vp;
+
gImpl.myPayloads [(int)payloadChar] = *impl;
gImpl.myPayloadLibraries [(int)payloadChar] = bridgeLib;
+ gImpl.myPayloadDestroyFuncs [(int)payloadChar] = destroyFunc;

if (!gDefaultPayload)
{
diff --git a/mama/c_cpp/src/c/mama/mama.h b/mama/c_cpp/src/c/mama/mama.h
index a574f36..07a9115 100644
--- a/mama/c_cpp/src/c/mama/mama.h
+++ b/mama/c_cpp/src/c/mama/mama.h
@@ -133,11 +133,34 @@ extern "C"
*/
MAMAExpDLL
extern mama_status
- mama_loadBridge (mamaBridge* impl, const char* middleware);
+ mama_loadBridge (mamaBridge* impl,
+ const char* middleware);

+ /**
+ * Load a payload bridge by its well-known library name string. E.g. "wmsg"
+ * will load the WombatMsg payload, provided it exists on the system.
+ *
+ * @param bridge The payload bridge object.
+ * @param payloadName The well-known name of the payload library.
+ */
+ MAMAExpDLL
+ extern mama_status
+ mama_loadPayloadBridge (mamaPayloadBridge* bridge,
+ const char* payloadName);
+
+ /**
+ * Load a payload bridge by using the mamaPayloadType identifier. This
+ * utilises mamaPayload_convertToLibString() in order to retrieve the
+ * well-known library string for the payload library, and this is then
+ * delegated to mama_loadPayloadBridge.
+ *
+ * @param bridge The payload bridge object.
+ * @param payload The payload enumeration identifier.
+ */
MAMAExpDLL
extern mama_status
- mama_loadPayloadBridge (mamaPayloadBridge* bridge, const char* payloadName);
+ mama_loadPayloadBridgeByType (mamaPayloadBridge* bridge,
+ mamaPayloadType payload);

/** Load the bridge specified by middleware string using the path specified by the user.
* If the bridge has already been loaded then the existing bridge instance
diff --git a/mama/c_cpp/src/c/mama/msg.h b/mama/c_cpp/src/c/mama/msg.h
index bf0d562..a9c3438 100644
--- a/mama/c_cpp/src/c/mama/msg.h
+++ b/mama/c_cpp/src/c/mama/msg.h
@@ -74,6 +74,36 @@ extern const char*
mamaPayload_convertToString (mamaPayloadType payloadType);

/**
+ * Convert a string to a mamaPayloadType.
+ *
+ * @param str The str to convert.
+ */
+MAMAExpDLL
+extern mamaPayloadType
+mamaPayload_convertFromString (const char* str);
+
+/**
+ * Convert a mamaPayloadType value to a well-known library name string
+ * for that payload. E.g. MAMA_PAYLOAD_WOMBAT_MSG translates into
+ * "wmsg" (of which the payload library would be libwmsgimpl.so on *nix).
+ *
+ * @param payloadType The payloadType to convert to a library name string.
+ */
+MAMAExpDLL
+extern const char*
+mamaPayload_convertToLibString (mamaPayloadType payloadType);
+
+/**
+ * Convert a well-known library name to a mamaPayloadType value
+ * for that payload. E.g. "wmsg" translates into MAMA_PAYLOAD_WOMBAT_MSG.
+ *
+ * @param str The library name string to convert to a mamaPayloadType.
+ */
+MAMAExpDLL
+extern mamaPayloadType
+mamaPayload_convertFromLibString (const char* str);
+
+/**
* Create a mamaMsg.
*
* @param msg The location of a mamaMsg where to store the result.
diff --git a/mama/c_cpp/src/c/msg.c b/mama/c_cpp/src/c/msg.c
index 286fcad..506c390 100644
--- a/mama/c_cpp/src/c/msg.c
+++ b/mama/c_cpp/src/c/msg.c
@@ -41,13 +41,39 @@

#include "wombat/wincompat.h"

-
#define MAX_SUBJECT 256

-#define NOFID 0
-
-static const int INCLUDE_FIELD_NAME = (1 << 4);
-static const int INCLUDE_FIELD_ID = (1 << 2);
+typedef struct PayloadTable
+{
+ mamaPayloadType mType;
+ const char* mName;
+ const char* mLibrary;
+}
+PayloadTable;
+
+static
+const PayloadTable kPayloads [] =
+{
+ {MAMA_PAYLOAD_ACTIV, "activ", "activmsg"},
+ {MAMA_PAYLOAD_AVIS, "AVIS", "avismsg"},
+ {MAMA_PAYLOAD_EXEGY, "EXEGY", "exegymsg"},
+ {MAMA_PAYLOAD_FAST, "FAST", "fastmsg"},
+ {MAMA_PAYLOAD_IBMWFO, "ibmwfo", "ibmwfomsg"},
+ {MAMA_PAYLOAD_INRUSH, "INRUSH", "inrushmsg"},
+ {MAMA_PAYLOAD_QPID, "QPID", "qpidmsg"},
+ {MAMA_PAYLOAD_RAI, "rai", "raimsg"},
+ {MAMA_PAYLOAD_SOLACE, "solacemsg", "solacemsg"},
+ {MAMA_PAYLOAD_TIBRV, "TIBRV", "tibrvmsg"},
+ {MAMA_PAYLOAD_TICK42BLP, "TICK42BLP", "tick42blpmsg"},
+ {MAMA_PAYLOAD_TICK42RMDS, "TICK42RMDS", "tick42rmdsmsg"},
+ {MAMA_PAYLOAD_UMS, "UMS", "umsmsg"},
+ {MAMA_PAYLOAD_V5, "V5", "wcachemsg"},
+ {MAMA_PAYLOAD_VULCAN, "Vulcan", "vulcanmsg"},
+ {MAMA_PAYLOAD_WOMBAT_MSG, "WombatMsg", "wmsg"},
+
+ /* Linear search requires unknown to be at the end-of-table */
+ {MAMA_PAYLOAD_UNKNOWN, "unknown", NULL}
+};

typedef struct mamaMsgIteratorImpl
{
@@ -106,6 +132,66 @@ typedef struct mamaMsgImpl_
static mama_status
mamaMsgImpl_destroyLastVectorMsg (mamaMsgImpl *impl);

+static
+const PayloadTable*
+mamaPayloadImpl_lookupTableByType (mamaPayloadType payloadType)
+{
+ int k = 0;
+ for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
+ {
+ if (kPayloads[k].mType == payloadType)
+ {
+ break;
+ }
+ }
+
+ return &kPayloads[k];
+}
+
+static
+const PayloadTable*
+mamaPayloadImpl_lookupTableByName (const char* name)
+{
+ int k = 0;
+
+ if (!name)
+ {
+ return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
+ }
+
+ for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
+ {
+ if (0 == strcasecmp (name, kPayloads[k].mName))
+ {
+ break;
+ }
+ }
+
+ return &kPayloads[k];
+}
+
+static
+const PayloadTable*
+mamaPayloadImpl_lookupTableByLibrary (const char* name)
+{
+ int k = 0;
+
+ if (!name)
+ {
+ return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
+ }
+
+ for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
+ {
+ if (0 == strcasecmp (name, kPayloads[k].mLibrary))
+ {
+ break;
+ }
+ }
+
+ return &kPayloads[k];
+}
+
/*=================================================================
= Public functions - defined in mama/msg.h
================================================================*/
@@ -166,7 +252,9 @@ mamaMsg_destroy (mamaMsg msg)
if (impl->mCurrentField)
{
mamaMsgField_destroy (impl->mCurrentField);
+ impl->mCurrentField = NULL;
}
+
/*Destroy the reusable field object*/
if (impl->mCopy)
{
@@ -366,44 +454,29 @@ mamaMsgImpl_getPayload (const mamaMsg msg, msgPayload* payload)
const char*
mamaPayload_convertToString (mamaPayloadType payloadType)
{
- switch (payloadType)
- {
- case MAMA_PAYLOAD_SOLACE:
- return "solacemsg";
- case MAMA_PAYLOAD_V5:
- return "V5";
- case MAMA_PAYLOAD_AVIS:
- return "AVIS";
- case MAMA_PAYLOAD_TICK42BLP:
- return "TICK42BLP";
- case MAMA_PAYLOAD_FAST:
- return "FAST";
- case MAMA_PAYLOAD_RAI:
- return "rai";
- case MAMA_PAYLOAD_UMS:
- return "UMS";
- case MAMA_PAYLOAD_TICK42RMDS:
- return "TICK42RMDS";
- case MAMA_PAYLOAD_QPID:
- return "QPID";
- case MAMA_PAYLOAD_TIBRV:
- return "TIBRV";
- case MAMA_PAYLOAD_IBMWFO:
- return "ibmwfo";
- case MAMA_PAYLOAD_ACTIV:
- return "activ";
- case MAMA_PAYLOAD_VULCAN:
- return "Vulcan";
- case MAMA_PAYLOAD_WOMBAT_MSG:
- return "WombatMsg";
- case MAMA_PAYLOAD_EXEGY:
- return "EXEGY";
- case MAMA_PAYLOAD_INRUSH:
- return "INRUSH";
-
- default:
- return "unknown";
- }
+ const PayloadTable* table = mamaPayloadImpl_lookupTableByType (payloadType);
+ return table->mName;
+}
+
+mamaPayloadType
+mamaPayload_convertFromString (const char* str)
+{
+ const PayloadTable* table = mamaPayloadImpl_lookupTableByName (str);
+ return table->mType;
+}
+
+const char*
+mamaPayload_convertToLibString (mamaPayloadType payloadType)
+{
+ const PayloadTable* table = mamaPayloadImpl_lookupTableByType (payloadType);
+ return table->mLibrary;
+}
+
+mamaPayloadType
+mamaPayload_convertFromLibString (const char* str)
+{
+ const PayloadTable* table = mamaPayloadImpl_lookupTableByLibrary (str);
+ return table->mType;
}

mama_status
@@ -3481,7 +3554,11 @@ mama_status mamaMsgIterator_destroy (mamaMsgIterator iterator)
}

if (impl->mCurrentField)
+ {
mamaMsgField_destroy (impl->mCurrentField);
+ impl->mCurrentField = NULL;
+ }
+
free (impl);
return MAMA_STATUS_OK;
}
diff --git a/mama/c_cpp/src/c/payloadbridge.h b/mama/c_cpp/src/c/payloadbridge.h
index 703c1a9..6ad7108 100644
--- a/mama/c_cpp/src/c/payloadbridge.h
+++ b/mama/c_cpp/src/c/payloadbridge.h
@@ -422,8 +422,12 @@ while(0) \
typedef mama_status
(*msgPayload_createImpl) (mamaPayloadBridge* result, char* identifier);

+typedef mama_status
+(*msgPayload_destroyImpl) (mamaPayloadBridge result);
+
typedef mamaPayloadType
(*msgPayload_getType) (void);
+
/*===================================================================
= msgPayload bridge function pointers =
====================================================================*/
diff --git a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
index ed18a64..1e14227 100644
--- a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
+++ b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
@@ -20,14 +20,9 @@ protected:

virtual void TearDown(void);

- typedef std::pair <mamaPayloadType, std::string> MamaPayloadPair;
- typedef std::map <mamaPayloadType, std::string> MamaPayloadMapType;
-
- typedef std::pair <mamaMiddleware, std::string> MamaMiddlewarePair;
- typedef std::map <mamaMiddleware, std::string> MamaMiddlewareMapType;
-
- MamaPayloadMapType payloadTestData;
- MamaMiddlewareMapType middlewareTestData;
+ typedef std::pair <std::string, std::string> PayloadNameLib;
+ typedef std::map <mamaPayloadType, PayloadNameLib> MamaPayloadMapType;
+ typedef std::map <mamaMiddleware, std::string> MamaMiddlewareMapType;

private:
void CreateTestData();
@@ -55,100 +50,81 @@ MamaEnumTestsC::TearDown()
void
MamaEnumTestsC::CreateTestData()
{
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_V5, "V5"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_AVIS, "AVIS"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42BLP, "TICK42BLP"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_FAST, "FAST"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_RAI, "rai"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UMS, "UMS"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42RMDS, "TICK42RMDS"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_QPID, "QPID"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TIBRV, "TIBRV"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_IBMWFO, "ibmwfo"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_ACTIV, "activ"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_VULCAN, "Vulcan"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_WOMBAT_MSG, "WombatMsg"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_EXEGY, "EXEGY"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_SOLACE, "solacemsg"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UNKNOWN, "unknown"));
-
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_WMW, "wmw"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_LBM, "lbm"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_TIBRV, "tibrv"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_AVIS, "AVIS"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_TICK42BLP, "tick42blp"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_SOLACE, "SOLACE"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_RAI, "rai"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_QPID, "QPID"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_EXEGY, "exegy"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_IBMWFO, "ibmwfo"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_ACTIV, "activ"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_TICK42RMDS, "tick42rmds"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_UMS, "ums"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_VULCAN, "vulcan"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_UNKNOWN, "unknown"));
}

-
-TEST_F (MamaEnumTestsC, testPayloadConvertToString)
+TEST_F (MamaEnumTestsC, testPayloadStringConversions)
{
+ MamaPayloadMapType testData;
+ testData[MAMA_PAYLOAD_ACTIV] = PayloadNameLib ("activ", "activmsg");
+ testData[MAMA_PAYLOAD_AVIS] = PayloadNameLib ("AVIS", "avismsg");
+ testData[MAMA_PAYLOAD_EXEGY] = PayloadNameLib ("EXEGY", "exegymsg");
+ testData[MAMA_PAYLOAD_FAST] = PayloadNameLib ("FAST", "fastmsg");
+ testData[MAMA_PAYLOAD_IBMWFO] = PayloadNameLib ("ibmwfo", "ibmwfomsg");
+ testData[MAMA_PAYLOAD_QPID] = PayloadNameLib ("QPID", "qpidmsg");
+ testData[MAMA_PAYLOAD_RAI] = PayloadNameLib ("rai", "raimsg");
+ testData[MAMA_PAYLOAD_SOLACE] = PayloadNameLib ("solacemsg", "solacemsg");
+ testData[MAMA_PAYLOAD_TIBRV] = PayloadNameLib ("TIBRV", "tibrvmsg");
+ testData[MAMA_PAYLOAD_TICK42BLP] = PayloadNameLib ("TICK42BLP", "tick42blpmsg");
+ testData[MAMA_PAYLOAD_TICK42RMDS] = PayloadNameLib ("TICK42RMDS", "tick42rmdsmsg");
+ testData[MAMA_PAYLOAD_UMS] = PayloadNameLib ("UMS", "umsmsg");
+ testData[MAMA_PAYLOAD_VULCAN] = PayloadNameLib ("Vulcan", "vulcanmsg");
+ testData[MAMA_PAYLOAD_WOMBAT_MSG] = PayloadNameLib ("WombatMsg", "wmsg");
+ testData[MAMA_PAYLOAD_V5] = PayloadNameLib ("V5", "wcachemsg");
+ testData[MAMA_PAYLOAD_UNKNOWN] = PayloadNameLib ("unknown", "");
+
MamaPayloadMapType::iterator itr;
- bool passed = true;

- for(itr = payloadTestData.begin(); itr != payloadTestData.end(); ++itr) {
+ for(itr = testData.begin(); itr != testData.end(); ++itr) {

- mamaPayloadType payload = (*itr).first;
- std::string expected = (*itr).second;
-
- std::string actual = mamaPayload_convertToString (payload);
-
- EXPECT_STREQ(actual.c_str(), expected.c_str());
-
- if (actual != expected)
- passed = false;
+ mamaPayloadType payload = itr->first;
+ PayloadNameLib& namelib = itr->second;
+ const char* name = namelib.first.c_str();
+ const char* library = namelib.second.c_str();
+
+ // Override for unknown, mama returns NULL for unknown library (so that
+ // it prevents libraries from trying to load libmamaunknownmsg).
+ if (MAMA_PAYLOAD_UNKNOWN == payload)
+ {
+ library = NULL;
+ }
+
+ // Check name/payload conversion
+ EXPECT_STREQ(name, mamaPayload_convertToString (payload));
+ EXPECT_EQ((char)payload, (char)mamaPayload_convertFromString (name));
+
+ // Check library/payload conversion
+ EXPECT_STREQ(library, mamaPayload_convertToLibString (payload));
+ EXPECT_EQ((char)payload, (char)mamaPayload_convertFromLibString (library));
}
-
- ASSERT_EQ( passed, true );
}

-TEST_F (MamaEnumTestsC, testMiddlewareConvertToString)
+TEST_F (MamaEnumTestsC, testMiddlewareStringConversions)
{
+ MamaMiddlewareMapType testData;
+ testData[MAMA_MIDDLEWARE_WMW] = "wmw";
+ testData[MAMA_MIDDLEWARE_LBM] = "lbm";
+ testData[MAMA_MIDDLEWARE_TIBRV] = "tibrv";
+ testData[MAMA_MIDDLEWARE_AVIS] = "AVIS";
+ testData[MAMA_MIDDLEWARE_TICK42BLP] = "tick42blp";
+ testData[MAMA_MIDDLEWARE_SOLACE] = "SOLACE";
+ testData[MAMA_MIDDLEWARE_RAI] = "rai";
+ testData[MAMA_MIDDLEWARE_QPID] = "QPID";
+ testData[MAMA_MIDDLEWARE_EXEGY] = "exegy";
+ testData[MAMA_MIDDLEWARE_IBMWFO] = "ibmwfo";
+ testData[MAMA_MIDDLEWARE_ACTIV] = "activ";
+ testData[MAMA_MIDDLEWARE_TICK42RMDS] = "tick42rmds";
+ testData[MAMA_MIDDLEWARE_UMS] = "ums";
+ testData[MAMA_MIDDLEWARE_VULCAN] = "vulcan";
+ testData[MAMA_MIDDLEWARE_UNKNOWN] = "unknown";
MamaMiddlewareMapType::iterator itr;
- bool passed = true;

- for(itr = middlewareTestData.begin(); itr != middlewareTestData.end(); ++itr) {
+ for(itr = testData.begin(); itr != testData.end(); ++itr) {

- mamaMiddleware middleware = (*itr).first;
- std::string expected = (*itr).second;
-
- std::string actual = mamaMiddleware_convertToString (middleware);
+ mamaMiddleware middleware = itr->first;
+ const char* name = itr->second.c_str();

- EXPECT_STREQ(actual.c_str(), expected.c_str());
-
- if (actual != expected)
- passed = false;
+ EXPECT_STREQ(name, mamaMiddleware_convertToString (middleware));
+ EXPECT_EQ((char)middleware, (char)mamaMiddleware_convertFromString (name));
}
-
- ASSERT_EQ (passed, true);
}

-TEST_F (MamaEnumTestsC, testMiddlewareConvertFromString)
-{
- MamaMiddlewareMapType::iterator itr;
- bool passed = true;
-
- for (itr = middlewareTestData.begin(); itr != middlewareTestData.end(); itr++) {
-
- mamaMiddleware expected = (*itr).first;
- std::string middleware = (*itr).second;
-
- mamaMiddleware actual = mamaMiddleware_convertFromString (middleware.c_str());
-
- EXPECT_EQ (actual, expected);
-
- if (actual != expected)
- passed = false;
- }
-
- ASSERT_EQ (passed, true);
-}
--
1.9.0


Lee Skillen <lskillen@...>
 

If a library hasn't been previously loaded then OpenMAMA will now attempt
to load it by converting the payload identifier into a well-known string
for the payload library. This behaviour is useful if the payload isn't
known ahead of time or if lazy loading is desired over enumeration of a
existing set of libraries. No changes made to existing API interfaces.

The following functions have been added :-

- mamaPayload_convertToLibString():
Maps a mamaPayloadType enumeration to the well-known library name.

- mamaPayload_convertFromLibString():
Inverse of mamaPayload_convertToLibString.

- mama_loadPayloadBridgeByType():
Takes a mamaPayloadType and utilises mamaPayload_convertToLibString()
to retrieve the library name for the payload, and then delegates this
to the existing mama_loadPayloadBridge() function.

The string conversions are implemented as a linear search, primarily due
to these functions not being on the hot path - Their use is prior to a
library being loaded and (I would imagine) would be seldom used in other
situations.

Additional changes :-

If msgPayload_destroyImpl has been exposed by the payload library then
this is now utilised to cleanup the payload upon shutdown properly. It
was observed that each payload always leaked memory due to lacking the
capability to destroy itself. Individual payloads will still need to
expose this - Backwards compatible as it won't be called otherwise.

The MamaEnumTestsC set of unit tests have been rewritten to utilise the
new conversion functionality. The tests have also been renamed as such:

- testPayloadToString -> testPayloadStringConversions
- testPayloadFromString -> testPayloadStringConversions
- testMiddlewareToString -> testMiddlewareStringConversions
- testMiddlewareFromString -> testMiddlewareStringConversions

Noticed that the ASSERT_EQ/EXPECT_EQ statements in gtest are sometimes
used incorrectly, in that the expected/actual are reversed (it should be
expected on the left and actual on the right).

Todo :-

Need to confirm well-known library names with all vendors - Will only be
important if JIT loading is actually used, but the names should be made
accurate anyway.

Signed-off-by: Lee Skillen <lskillen@...>
---
mama/c_cpp/src/c/mama.c | 77 ++++++++--
mama/c_cpp/src/c/mama/mama.h | 27 +++-
mama/c_cpp/src/c/mama/msg.h | 30 ++++
mama/c_cpp/src/c/msg.c | 163 +++++++++++++++------
mama/c_cpp/src/c/payloadbridge.h | 4 +
.../src/gunittest/c/payloadmiddlewareidtest.cpp | 152 ++++++++-----------
6 files changed, 304 insertions(+), 149 deletions(-)

diff --git a/mama/c_cpp/src/c/mama.c b/mama/c_cpp/src/c/mama.c
index 33a0768..3a7d7be 100644
--- a/mama/c_cpp/src/c/mama.c
+++ b/mama/c_cpp/src/c/mama.c
@@ -28,6 +28,7 @@
#include "wombat/strutils.h"

#include <mama/mama.h>
+#include <mama/msg.h>
#include <mama/error.h>
#include <mamainternal.h>
#include <mama/version.h>
@@ -168,6 +169,7 @@ typedef struct mamaImpl_
mamaPayloadBridge myPayloads[MAMA_PAYLOAD_MAX];
LIB_HANDLE myBridgeLibraries[MAMA_MIDDLEWARE_MAX];
LIB_HANDLE myPayloadLibraries[MAMA_PAYLOAD_MAX];
+ msgPayload_destroyImpl myPayloadDestroyFuncs[MAMA_PAYLOAD_MAX];
unsigned int myRefCount;
wthread_static_mutex_t myLock;
} mamaImpl;
@@ -175,7 +177,7 @@ typedef struct mamaImpl_
static mamaApplicationContext appContext;
static char mama_ver_string[256];

-static mamaImpl gImpl = {{0}, {0}, {0}, {0}, 0, WSTATIC_MUTEX_INITIALIZER};
+static mamaImpl gImpl = {{0}, {0}, {0}, {0}, {0}, 0, WSTATIC_MUTEX_INITIALIZER};

/* ************************************************************************* */
/* Private Function Prototypes. */
@@ -624,10 +626,21 @@ mamaInternal_findBridge ()
mamaPayloadBridge
mamaInternal_findPayload (char id)
{
+ mamaPayloadBridge bridge = NULL;
+
if ('\0' == id)
return NULL;

- return gImpl.myPayloads[(uint8_t)id];
+ bridge = gImpl.myPayloads[(uint8_t)id];
+
+ if (NULL == bridge)
+ {
+ /* Dynamically load the payload bridge just-in-time, if it hasn't
+ * already been loaded previously. */
+ mama_loadPayloadBridgeByType (&bridge, (mamaPayloadType)id);
+ }
+
+ return bridge;
}

mamaPayloadBridge
@@ -1241,17 +1254,33 @@ mama_closeCount (unsigned int* count)

cleanupReservedFields();

- /* Look for a bridge for each of the payloads and close them */
+ /* Look for a bridge for each of the payloads and close them */
for (payload = 0; payload != MAMA_PAYLOAD_MAX; ++payload)
{
- /* mamaPayloadBridgeImpl* impl = (mamaPayloadBridgeImpl*)
- * gImpl.myPayloads [(uint8_t)payload];*/
- gImpl.myPayloads[(uint8_t)payload] = NULL;
- if(gImpl.myPayloadLibraries[(uint8_t)payload])
+ uint8_t id = (uint8_t)payload;
+ LIB_HANDLE lib = gImpl.myPayloadLibraries[id];
+ mamaPayloadBridgeImpl* impl = gImpl.myPayloads[id];
+
+ if (!lib)
{
- closeSharedLib (gImpl.myPayloadLibraries[(uint8_t)payload]);
- gImpl.myPayloadLibraries[(uint8_t)payload] = NULL;
+ continue;
+ }
+
+ if (impl)
+ {
+ msgPayload_destroyImpl destroyFunc =
+ gImpl.myPayloadDestroyFuncs[id];
+
+ if (destroyFunc)
+ {
+ destroyFunc (impl);
+ }
+
+ gImpl.myPayloads[id] = NULL;
}
+
+ closeSharedLib (lib);
+ gImpl.myPayloadLibraries[id] = NULL;
}

gDefaultPayload = NULL;
@@ -1827,18 +1856,29 @@ mama_setDefaultPayload (char id)

mama_status
mama_loadPayloadBridge (mamaPayloadBridge* impl,
- const char* payloadName)
+ const char* payloadName)
{
return mama_loadPayloadBridgeInternal (impl, payloadName);
}
+
+mama_status
+mama_loadPayloadBridgeByType (mamaPayloadBridge* impl,
+ mamaPayloadType payload)
+{
+ const char* name = mamaPayload_convertToLibString (payload);
+ return mama_loadPayloadBridgeInternal (impl, name);
+}
+
mama_status
mama_loadPayloadBridgeInternal (mamaPayloadBridge* impl,
const char* payloadName)
{
char bridgeImplName [256];
char initFuncName [256];
+ char destroyFuncName [256];
LIB_HANDLE bridgeLib = NULL;
msgPayload_createImpl initFunc = NULL;
+ msgPayload_destroyImpl destroyFunc = NULL;
mama_status status = MAMA_STATUS_OK;
char payloadChar ='\0';
void* vp = NULL;
@@ -1865,11 +1905,10 @@ mama_loadPayloadBridgeInternal (mamaPayloadBridge* impl,
return MAMA_STATUS_NO_BRIDGE_IMPL;
}

- snprintf (initFuncName, 256, "%sPayload_createImpl", payloadName);
-
/* Gives a warning - casting from void* to bridge_createImpl func */
- vp = loadLibFunc (bridgeLib, initFuncName);
- initFunc = *(msgPayload_createImpl*) &vp;
+ snprintf (initFuncName, 256, "%sPayload_createImpl", payloadName);
+ vp = loadLibFunc (bridgeLib, initFuncName);
+ initFunc = *(msgPayload_createImpl*) &vp;

if (!initFunc)
{
@@ -1887,7 +1926,7 @@ mama_loadPayloadBridgeInternal (mamaPayloadBridge* impl,

if (MAMA_STATUS_OK != (status = initFunc (impl, &payloadChar)))
{
- wthread_static_mutex_unlock (&gImpl.myLock);
+ wthread_static_mutex_unlock (&gImpl.myLock);

return status;
}
@@ -1910,13 +1949,19 @@ mama_loadPayloadBridgeInternal (mamaPayloadBridge* impl,
"Payload bridge %s already loaded",
payloadName);

- wthread_static_mutex_unlock (&gImpl.myLock);
+ wthread_static_mutex_unlock (&gImpl.myLock);

return MAMA_STATUS_OK;
}

+ /* Gives a warning - casting from void* to bridge_destroyImpl func */
+ snprintf (destroyFuncName, 256, "%sPayload_destroyImpl", payloadName);
+ vp = loadLibFunc (bridgeLib, destroyFuncName);
+ destroyFunc = *(msgPayload_destroyImpl*) &vp;
+
gImpl.myPayloads [(int)payloadChar] = *impl;
gImpl.myPayloadLibraries [(int)payloadChar] = bridgeLib;
+ gImpl.myPayloadDestroyFuncs [(int)payloadChar] = destroyFunc;

if (!gDefaultPayload)
{
diff --git a/mama/c_cpp/src/c/mama/mama.h b/mama/c_cpp/src/c/mama/mama.h
index a574f36..07a9115 100644
--- a/mama/c_cpp/src/c/mama/mama.h
+++ b/mama/c_cpp/src/c/mama/mama.h
@@ -133,11 +133,34 @@ extern "C"
*/
MAMAExpDLL
extern mama_status
- mama_loadBridge (mamaBridge* impl, const char* middleware);
+ mama_loadBridge (mamaBridge* impl,
+ const char* middleware);

+ /**
+ * Load a payload bridge by its well-known library name string. E.g. "wmsg"
+ * will load the WombatMsg payload, provided it exists on the system.
+ *
+ * @param bridge The payload bridge object.
+ * @param payloadName The well-known name of the payload library.
+ */
+ MAMAExpDLL
+ extern mama_status
+ mama_loadPayloadBridge (mamaPayloadBridge* bridge,
+ const char* payloadName);
+
+ /**
+ * Load a payload bridge by using the mamaPayloadType identifier. This
+ * utilises mamaPayload_convertToLibString() in order to retrieve the
+ * well-known library string for the payload library, and this is then
+ * delegated to mama_loadPayloadBridge.
+ *
+ * @param bridge The payload bridge object.
+ * @param payload The payload enumeration identifier.
+ */
MAMAExpDLL
extern mama_status
- mama_loadPayloadBridge (mamaPayloadBridge* bridge, const char* payloadName);
+ mama_loadPayloadBridgeByType (mamaPayloadBridge* bridge,
+ mamaPayloadType payload);

/** Load the bridge specified by middleware string using the path specified by the user.
* If the bridge has already been loaded then the existing bridge instance
diff --git a/mama/c_cpp/src/c/mama/msg.h b/mama/c_cpp/src/c/mama/msg.h
index bf0d562..a9c3438 100644
--- a/mama/c_cpp/src/c/mama/msg.h
+++ b/mama/c_cpp/src/c/mama/msg.h
@@ -74,6 +74,36 @@ extern const char*
mamaPayload_convertToString (mamaPayloadType payloadType);

/**
+ * Convert a string to a mamaPayloadType.
+ *
+ * @param str The str to convert.
+ */
+MAMAExpDLL
+extern mamaPayloadType
+mamaPayload_convertFromString (const char* str);
+
+/**
+ * Convert a mamaPayloadType value to a well-known library name string
+ * for that payload. E.g. MAMA_PAYLOAD_WOMBAT_MSG translates into
+ * "wmsg" (of which the payload library would be libwmsgimpl.so on *nix).
+ *
+ * @param payloadType The payloadType to convert to a library name string.
+ */
+MAMAExpDLL
+extern const char*
+mamaPayload_convertToLibString (mamaPayloadType payloadType);
+
+/**
+ * Convert a well-known library name to a mamaPayloadType value
+ * for that payload. E.g. "wmsg" translates into MAMA_PAYLOAD_WOMBAT_MSG.
+ *
+ * @param str The library name string to convert to a mamaPayloadType.
+ */
+MAMAExpDLL
+extern mamaPayloadType
+mamaPayload_convertFromLibString (const char* str);
+
+/**
* Create a mamaMsg.
*
* @param msg The location of a mamaMsg where to store the result.
diff --git a/mama/c_cpp/src/c/msg.c b/mama/c_cpp/src/c/msg.c
index 286fcad..506c390 100644
--- a/mama/c_cpp/src/c/msg.c
+++ b/mama/c_cpp/src/c/msg.c
@@ -41,13 +41,39 @@

#include "wombat/wincompat.h"

-
#define MAX_SUBJECT 256

-#define NOFID 0
-
-static const int INCLUDE_FIELD_NAME = (1 << 4);
-static const int INCLUDE_FIELD_ID = (1 << 2);
+typedef struct PayloadTable
+{
+ mamaPayloadType mType;
+ const char* mName;
+ const char* mLibrary;
+}
+PayloadTable;
+
+static
+const PayloadTable kPayloads [] =
+{
+ {MAMA_PAYLOAD_ACTIV, "activ", "activmsg"},
+ {MAMA_PAYLOAD_AVIS, "AVIS", "avismsg"},
+ {MAMA_PAYLOAD_EXEGY, "EXEGY", "exegymsg"},
+ {MAMA_PAYLOAD_FAST, "FAST", "fastmsg"},
+ {MAMA_PAYLOAD_IBMWFO, "ibmwfo", "ibmwfomsg"},
+ {MAMA_PAYLOAD_INRUSH, "INRUSH", "inrushmsg"},
+ {MAMA_PAYLOAD_QPID, "QPID", "qpidmsg"},
+ {MAMA_PAYLOAD_RAI, "rai", "raimsg"},
+ {MAMA_PAYLOAD_SOLACE, "solacemsg", "solacemsg"},
+ {MAMA_PAYLOAD_TIBRV, "TIBRV", "tibrvmsg"},
+ {MAMA_PAYLOAD_TICK42BLP, "TICK42BLP", "tick42blpmsg"},
+ {MAMA_PAYLOAD_TICK42RMDS, "TICK42RMDS", "tick42rmdsmsg"},
+ {MAMA_PAYLOAD_UMS, "UMS", "umsmsg"},
+ {MAMA_PAYLOAD_V5, "V5", "wcachemsg"},
+ {MAMA_PAYLOAD_VULCAN, "Vulcan", "vulcanmsg"},
+ {MAMA_PAYLOAD_WOMBAT_MSG, "WombatMsg", "wmsg"},
+
+ /* Linear search requires unknown to be at the end-of-table */
+ {MAMA_PAYLOAD_UNKNOWN, "unknown", NULL}
+};

typedef struct mamaMsgIteratorImpl
{
@@ -106,6 +132,66 @@ typedef struct mamaMsgImpl_
static mama_status
mamaMsgImpl_destroyLastVectorMsg (mamaMsgImpl *impl);

+static
+const PayloadTable*
+mamaPayloadImpl_lookupTableByType (mamaPayloadType payloadType)
+{
+ int k = 0;
+ for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
+ {
+ if (kPayloads[k].mType == payloadType)
+ {
+ break;
+ }
+ }
+
+ return &kPayloads[k];
+}
+
+static
+const PayloadTable*
+mamaPayloadImpl_lookupTableByName (const char* name)
+{
+ int k = 0;
+
+ if (!name)
+ {
+ return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
+ }
+
+ for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
+ {
+ if (0 == strcasecmp (name, kPayloads[k].mName))
+ {
+ break;
+ }
+ }
+
+ return &kPayloads[k];
+}
+
+static
+const PayloadTable*
+mamaPayloadImpl_lookupTableByLibrary (const char* name)
+{
+ int k = 0;
+
+ if (!name)
+ {
+ return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
+ }
+
+ for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
+ {
+ if (0 == strcasecmp (name, kPayloads[k].mLibrary))
+ {
+ break;
+ }
+ }
+
+ return &kPayloads[k];
+}
+
/*=================================================================
= Public functions - defined in mama/msg.h
================================================================*/
@@ -166,7 +252,9 @@ mamaMsg_destroy (mamaMsg msg)
if (impl->mCurrentField)
{
mamaMsgField_destroy (impl->mCurrentField);
+ impl->mCurrentField = NULL;
}
+
/*Destroy the reusable field object*/
if (impl->mCopy)
{
@@ -366,44 +454,29 @@ mamaMsgImpl_getPayload (const mamaMsg msg, msgPayload* payload)
const char*
mamaPayload_convertToString (mamaPayloadType payloadType)
{
- switch (payloadType)
- {
- case MAMA_PAYLOAD_SOLACE:
- return "solacemsg";
- case MAMA_PAYLOAD_V5:
- return "V5";
- case MAMA_PAYLOAD_AVIS:
- return "AVIS";
- case MAMA_PAYLOAD_TICK42BLP:
- return "TICK42BLP";
- case MAMA_PAYLOAD_FAST:
- return "FAST";
- case MAMA_PAYLOAD_RAI:
- return "rai";
- case MAMA_PAYLOAD_UMS:
- return "UMS";
- case MAMA_PAYLOAD_TICK42RMDS:
- return "TICK42RMDS";
- case MAMA_PAYLOAD_QPID:
- return "QPID";
- case MAMA_PAYLOAD_TIBRV:
- return "TIBRV";
- case MAMA_PAYLOAD_IBMWFO:
- return "ibmwfo";
- case MAMA_PAYLOAD_ACTIV:
- return "activ";
- case MAMA_PAYLOAD_VULCAN:
- return "Vulcan";
- case MAMA_PAYLOAD_WOMBAT_MSG:
- return "WombatMsg";
- case MAMA_PAYLOAD_EXEGY:
- return "EXEGY";
- case MAMA_PAYLOAD_INRUSH:
- return "INRUSH";
-
- default:
- return "unknown";
- }
+ const PayloadTable* table = mamaPayloadImpl_lookupTableByType (payloadType);
+ return table->mName;
+}
+
+mamaPayloadType
+mamaPayload_convertFromString (const char* str)
+{
+ const PayloadTable* table = mamaPayloadImpl_lookupTableByName (str);
+ return table->mType;
+}
+
+const char*
+mamaPayload_convertToLibString (mamaPayloadType payloadType)
+{
+ const PayloadTable* table = mamaPayloadImpl_lookupTableByType (payloadType);
+ return table->mLibrary;
+}
+
+mamaPayloadType
+mamaPayload_convertFromLibString (const char* str)
+{
+ const PayloadTable* table = mamaPayloadImpl_lookupTableByLibrary (str);
+ return table->mType;
}

mama_status
@@ -3481,7 +3554,11 @@ mama_status mamaMsgIterator_destroy (mamaMsgIterator iterator)
}

if (impl->mCurrentField)
+ {
mamaMsgField_destroy (impl->mCurrentField);
+ impl->mCurrentField = NULL;
+ }
+
free (impl);
return MAMA_STATUS_OK;
}
diff --git a/mama/c_cpp/src/c/payloadbridge.h b/mama/c_cpp/src/c/payloadbridge.h
index 703c1a9..6ad7108 100644
--- a/mama/c_cpp/src/c/payloadbridge.h
+++ b/mama/c_cpp/src/c/payloadbridge.h
@@ -422,8 +422,12 @@ while(0) \
typedef mama_status
(*msgPayload_createImpl) (mamaPayloadBridge* result, char* identifier);

+typedef mama_status
+(*msgPayload_destroyImpl) (mamaPayloadBridge result);
+
typedef mamaPayloadType
(*msgPayload_getType) (void);
+
/*===================================================================
= msgPayload bridge function pointers =
====================================================================*/
diff --git a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
index ed18a64..1e14227 100644
--- a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
+++ b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
@@ -20,14 +20,9 @@ protected:

virtual void TearDown(void);

- typedef std::pair <mamaPayloadType, std::string> MamaPayloadPair;
- typedef std::map <mamaPayloadType, std::string> MamaPayloadMapType;
-
- typedef std::pair <mamaMiddleware, std::string> MamaMiddlewarePair;
- typedef std::map <mamaMiddleware, std::string> MamaMiddlewareMapType;
-
- MamaPayloadMapType payloadTestData;
- MamaMiddlewareMapType middlewareTestData;
+ typedef std::pair <std::string, std::string> PayloadNameLib;
+ typedef std::map <mamaPayloadType, PayloadNameLib> MamaPayloadMapType;
+ typedef std::map <mamaMiddleware, std::string> MamaMiddlewareMapType;

private:
void CreateTestData();
@@ -55,100 +50,81 @@ MamaEnumTestsC::TearDown()
void
MamaEnumTestsC::CreateTestData()
{
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_V5, "V5"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_AVIS, "AVIS"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42BLP, "TICK42BLP"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_FAST, "FAST"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_RAI, "rai"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UMS, "UMS"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42RMDS, "TICK42RMDS"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_QPID, "QPID"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TIBRV, "TIBRV"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_IBMWFO, "ibmwfo"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_ACTIV, "activ"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_VULCAN, "Vulcan"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_WOMBAT_MSG, "WombatMsg"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_EXEGY, "EXEGY"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_SOLACE, "solacemsg"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UNKNOWN, "unknown"));
-
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_WMW, "wmw"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_LBM, "lbm"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_TIBRV, "tibrv"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_AVIS, "AVIS"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_TICK42BLP, "tick42blp"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_SOLACE, "SOLACE"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_RAI, "rai"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_QPID, "QPID"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_EXEGY, "exegy"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_IBMWFO, "ibmwfo"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_ACTIV, "activ"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_TICK42RMDS, "tick42rmds"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_UMS, "ums"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_VULCAN, "vulcan"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_UNKNOWN, "unknown"));
}

-
-TEST_F (MamaEnumTestsC, testPayloadConvertToString)
+TEST_F (MamaEnumTestsC, testPayloadStringConversions)
{
+ MamaPayloadMapType testData;
+ testData[MAMA_PAYLOAD_ACTIV] = PayloadNameLib ("activ", "activmsg");
+ testData[MAMA_PAYLOAD_AVIS] = PayloadNameLib ("AVIS", "avismsg");
+ testData[MAMA_PAYLOAD_EXEGY] = PayloadNameLib ("EXEGY", "exegymsg");
+ testData[MAMA_PAYLOAD_FAST] = PayloadNameLib ("FAST", "fastmsg");
+ testData[MAMA_PAYLOAD_IBMWFO] = PayloadNameLib ("ibmwfo", "ibmwfomsg");
+ testData[MAMA_PAYLOAD_QPID] = PayloadNameLib ("QPID", "qpidmsg");
+ testData[MAMA_PAYLOAD_RAI] = PayloadNameLib ("rai", "raimsg");
+ testData[MAMA_PAYLOAD_SOLACE] = PayloadNameLib ("solacemsg", "solacemsg");
+ testData[MAMA_PAYLOAD_TIBRV] = PayloadNameLib ("TIBRV", "tibrvmsg");
+ testData[MAMA_PAYLOAD_TICK42BLP] = PayloadNameLib ("TICK42BLP", "tick42blpmsg");
+ testData[MAMA_PAYLOAD_TICK42RMDS] = PayloadNameLib ("TICK42RMDS", "tick42rmdsmsg");
+ testData[MAMA_PAYLOAD_UMS] = PayloadNameLib ("UMS", "umsmsg");
+ testData[MAMA_PAYLOAD_VULCAN] = PayloadNameLib ("Vulcan", "vulcanmsg");
+ testData[MAMA_PAYLOAD_WOMBAT_MSG] = PayloadNameLib ("WombatMsg", "wmsg");
+ testData[MAMA_PAYLOAD_V5] = PayloadNameLib ("V5", "wcachemsg");
+ testData[MAMA_PAYLOAD_UNKNOWN] = PayloadNameLib ("unknown", "");
+
MamaPayloadMapType::iterator itr;
- bool passed = true;

- for(itr = payloadTestData.begin(); itr != payloadTestData.end(); ++itr) {
+ for(itr = testData.begin(); itr != testData.end(); ++itr) {

- mamaPayloadType payload = (*itr).first;
- std::string expected = (*itr).second;
-
- std::string actual = mamaPayload_convertToString (payload);
-
- EXPECT_STREQ(actual.c_str(), expected.c_str());
-
- if (actual != expected)
- passed = false;
+ mamaPayloadType payload = itr->first;
+ PayloadNameLib& namelib = itr->second;
+ const char* name = namelib.first.c_str();
+ const char* library = namelib.second.c_str();
+
+ // Override for unknown, mama returns NULL for unknown library (so that
+ // it prevents libraries from trying to load libmamaunknownmsg).
+ if (MAMA_PAYLOAD_UNKNOWN == payload)
+ {
+ library = NULL;
+ }
+
+ // Check name/payload conversion
+ EXPECT_STREQ(name, mamaPayload_convertToString (payload));
+ EXPECT_EQ((char)payload, (char)mamaPayload_convertFromString (name));
+
+ // Check library/payload conversion
+ EXPECT_STREQ(library, mamaPayload_convertToLibString (payload));
+ EXPECT_EQ((char)payload, (char)mamaPayload_convertFromLibString (library));
}
-
- ASSERT_EQ( passed, true );
}

-TEST_F (MamaEnumTestsC, testMiddlewareConvertToString)
+TEST_F (MamaEnumTestsC, testMiddlewareStringConversions)
{
+ MamaMiddlewareMapType testData;
+ testData[MAMA_MIDDLEWARE_WMW] = "wmw";
+ testData[MAMA_MIDDLEWARE_LBM] = "lbm";
+ testData[MAMA_MIDDLEWARE_TIBRV] = "tibrv";
+ testData[MAMA_MIDDLEWARE_AVIS] = "AVIS";
+ testData[MAMA_MIDDLEWARE_TICK42BLP] = "tick42blp";
+ testData[MAMA_MIDDLEWARE_SOLACE] = "SOLACE";
+ testData[MAMA_MIDDLEWARE_RAI] = "rai";
+ testData[MAMA_MIDDLEWARE_QPID] = "QPID";
+ testData[MAMA_MIDDLEWARE_EXEGY] = "exegy";
+ testData[MAMA_MIDDLEWARE_IBMWFO] = "ibmwfo";
+ testData[MAMA_MIDDLEWARE_ACTIV] = "activ";
+ testData[MAMA_MIDDLEWARE_TICK42RMDS] = "tick42rmds";
+ testData[MAMA_MIDDLEWARE_UMS] = "ums";
+ testData[MAMA_MIDDLEWARE_VULCAN] = "vulcan";
+ testData[MAMA_MIDDLEWARE_UNKNOWN] = "unknown";
MamaMiddlewareMapType::iterator itr;
- bool passed = true;

- for(itr = middlewareTestData.begin(); itr != middlewareTestData.end(); ++itr) {
+ for(itr = testData.begin(); itr != testData.end(); ++itr) {

- mamaMiddleware middleware = (*itr).first;
- std::string expected = (*itr).second;
-
- std::string actual = mamaMiddleware_convertToString (middleware);
+ mamaMiddleware middleware = itr->first;
+ const char* name = itr->second.c_str();

- EXPECT_STREQ(actual.c_str(), expected.c_str());
-
- if (actual != expected)
- passed = false;
+ EXPECT_STREQ(name, mamaMiddleware_convertToString (middleware));
+ EXPECT_EQ((char)middleware, (char)mamaMiddleware_convertFromString (name));
}
-
- ASSERT_EQ (passed, true);
}

-TEST_F (MamaEnumTestsC, testMiddlewareConvertFromString)
-{
- MamaMiddlewareMapType::iterator itr;
- bool passed = true;
-
- for (itr = middlewareTestData.begin(); itr != middlewareTestData.end(); itr++) {
-
- mamaMiddleware expected = (*itr).first;
- std::string middleware = (*itr).second;
-
- mamaMiddleware actual = mamaMiddleware_convertFromString (middleware.c_str());
-
- EXPECT_EQ (actual, expected);
-
- if (actual != expected)
- passed = false;
- }
-
- ASSERT_EQ (passed, true);
-}


Damian Maguire <DMaguire@...>
 

Cheers for this Lee, looks very interesting. From a quick look/preliminary
test the changes look solid, but given the size and scope of the change I
think the best thing to do is fire it into it's own branch for a while and
let people check it out for themselves. As such I'm going to stick it into
a feature-jit-payload branch on the main repo. Anyone who's interested can
then have a look and comment on the changes, and we can fix any bugs there
as required.

Regarding testing these changes, have you guys actually got this running
with any of the open source bridges, or do you have any sample testing you
can show us to demonstrate that it both works, and doesn't break the
existing code?

Just a quick note on the patch files you've attached - they seem to have
landed (to me at least) with DOS line endings, which messes with the git
apply step. It's not a big deal, since I can easily enough strip the line
endings, but probably something to watch out for with future submissions
(particularly if you touch any of the unfortunate files that already mix
and match line endings).

I'm also taking a look at the other patches you've sent through, so I'll
get back to you on those.

Cheers,

Damian



On 2/19/14 3:53 PM, "Lee Skillen" <lskillen@...> wrote:


If a library hasn't been previously loaded then OpenMAMA will now attempt
to load it by converting the payload identifier into a well-known string
for the payload library. This behaviour is useful if the payload isn't
known ahead of time or if lazy loading is desired over enumeration of a
existing set of libraries. No changes made to existing API interfaces.

The following functions have been added :-

- mamaPayload_convertToLibString():
Maps a mamaPayloadType enumeration to the well-known library name.

- mamaPayload_convertFromLibString():
Inverse of mamaPayload_convertToLibString.

- mama_loadPayloadBridgeByType():
Takes a mamaPayloadType and utilises mamaPayload_convertToLibString()
to retrieve the library name for the payload, and then delegates this
to the existing mama_loadPayloadBridge() function.

The string conversions are implemented as a linear search, primarily due
to these functions not being on the hot path - Their use is prior to a
library being loaded and (I would imagine) would be seldom used in other
situations.

Additional changes :-

If msgPayload_destroyImpl has been exposed by the payload library then
this is now utilised to cleanup the payload upon shutdown properly. It
was observed that each payload always leaked memory due to lacking the
capability to destroy itself. Individual payloads will still need to
expose this - Backwards compatible as it won't be called otherwise.

The MamaEnumTestsC set of unit tests have been rewritten to utilise the
new conversion functionality. The tests have also been renamed as such:

- testPayloadToString -> testPayloadStringConversions
- testPayloadFromString -> testPayloadStringConversions
- testMiddlewareToString -> testMiddlewareStringConversions
- testMiddlewareFromString -> testMiddlewareStringConversions

Noticed that the ASSERT_EQ/EXPECT_EQ statements in gtest are sometimes
used incorrectly, in that the expected/actual are reversed (it should be
expected on the left and actual on the right).

Todo :-

Need to confirm well-known library names with all vendors - Will only be
important if JIT loading is actually used, but the names should be made
accurate anyway.

Signed-off-by: Lee Skillen <lskillen@...>
---
mama/c_cpp/src/c/mama.c | 77 ++++++++--
mama/c_cpp/src/c/mama/mama.h | 27 +++-
mama/c_cpp/src/c/mama/msg.h | 30 ++++
mama/c_cpp/src/c/msg.c | 163
+++++++++++++++------
mama/c_cpp/src/c/payloadbridge.h | 4 +
.../src/gunittest/c/payloadmiddlewareidtest.cpp | 152
++++++++-----------
6 files changed, 304 insertions(+), 149 deletions(-)

diff --git a/mama/c_cpp/src/c/mama.c b/mama/c_cpp/src/c/mama.c
index 33a0768..3a7d7be 100644
--- a/mama/c_cpp/src/c/mama.c
+++ b/mama/c_cpp/src/c/mama.c
@@ -28,6 +28,7 @@
#include "wombat/strutils.h"

#include <mama/mama.h>
+#include <mama/msg.h>
#include <mama/error.h>
#include <mamainternal.h>
#include <mama/version.h>
@@ -168,6 +169,7 @@ typedef struct mamaImpl_
mamaPayloadBridge myPayloads[MAMA_PAYLOAD_MAX];
LIB_HANDLE myBridgeLibraries[MAMA_MIDDLEWARE_MAX];
LIB_HANDLE myPayloadLibraries[MAMA_PAYLOAD_MAX];
+ msgPayload_destroyImpl myPayloadDestroyFuncs[MAMA_PAYLOAD_MAX];
unsigned int myRefCount;
wthread_static_mutex_t myLock;
} mamaImpl;
@@ -175,7 +177,7 @@ typedef struct mamaImpl_
static mamaApplicationContext appContext;
static char mama_ver_string[256];

-static mamaImpl gImpl = {{0}, {0}, {0}, {0}, 0,
WSTATIC_MUTEX_INITIALIZER};
+static mamaImpl gImpl = {{0}, {0}, {0}, {0}, {0}, 0,
WSTATIC_MUTEX_INITIALIZER};

/*
*************************************************************************
*/
/* Private Function Prototypes. */
@@ -624,10 +626,21 @@ mamaInternal_findBridge ()
mamaPayloadBridge
mamaInternal_findPayload (char id)
{
+ mamaPayloadBridge bridge = NULL;
+
if ('\0' == id)
return NULL;

- return gImpl.myPayloads[(uint8_t)id];
+ bridge = gImpl.myPayloads[(uint8_t)id];
+
+ if (NULL == bridge)
+ {
+ /* Dynamically load the payload bridge just-in-time, if it
hasn't
+ * already been loaded previously. */
+ mama_loadPayloadBridgeByType (&bridge, (mamaPayloadType)id);
+ }
+
+ return bridge;
}

mamaPayloadBridge
@@ -1241,17 +1254,33 @@ mama_closeCount (unsigned int* count)

cleanupReservedFields();

- /* Look for a bridge for each of the payloads and close them */
+ /* Look for a bridge for each of the payloads and close them */
for (payload = 0; payload != MAMA_PAYLOAD_MAX; ++payload)
{
- /* mamaPayloadBridgeImpl* impl = (mamaPayloadBridgeImpl*)
- * gImpl.myPayloads [(uint8_t)payload];*/
- gImpl.myPayloads[(uint8_t)payload] = NULL;
- if(gImpl.myPayloadLibraries[(uint8_t)payload])
+ uint8_t id = (uint8_t)payload;
+ LIB_HANDLE lib = gImpl.myPayloadLibraries[id];
+ mamaPayloadBridgeImpl* impl = gImpl.myPayloads[id];
+
+ if (!lib)
{
- closeSharedLib
(gImpl.myPayloadLibraries[(uint8_t)payload]);
- gImpl.myPayloadLibraries[(uint8_t)payload] = NULL;
+ continue;
+ }
+
+ if (impl)
+ {
+ msgPayload_destroyImpl destroyFunc =
+ gImpl.myPayloadDestroyFuncs[id];
+
+ if (destroyFunc)
+ {
+ destroyFunc (impl);
+ }
+
+ gImpl.myPayloads[id] = NULL;
}
+
+ closeSharedLib (lib);
+ gImpl.myPayloadLibraries[id] = NULL;
}

gDefaultPayload = NULL;
@@ -1827,18 +1856,29 @@ mama_setDefaultPayload (char id)

mama_status
mama_loadPayloadBridge (mamaPayloadBridge* impl,
- const char* payloadName)
+ const char* payloadName)
{
return mama_loadPayloadBridgeInternal (impl, payloadName);
}
+
+mama_status
+mama_loadPayloadBridgeByType (mamaPayloadBridge* impl,
+ mamaPayloadType payload)
+{
+ const char* name = mamaPayload_convertToLibString (payload);
+ return mama_loadPayloadBridgeInternal (impl, name);
+}
+
mama_status
mama_loadPayloadBridgeInternal (mamaPayloadBridge* impl,
const char* payloadName)
{
char bridgeImplName [256];
char initFuncName [256];
+ char destroyFuncName [256];
LIB_HANDLE bridgeLib = NULL;
msgPayload_createImpl initFunc = NULL;
+ msgPayload_destroyImpl destroyFunc = NULL;
mama_status status = MAMA_STATUS_OK;
char payloadChar ='\0';
void* vp = NULL;
@@ -1865,11 +1905,10 @@ mama_loadPayloadBridgeInternal
(mamaPayloadBridge* impl,
return MAMA_STATUS_NO_BRIDGE_IMPL;
}

- snprintf (initFuncName, 256, "%sPayload_createImpl", payloadName);
-
/* Gives a warning - casting from void* to bridge_createImpl func */
- vp = loadLibFunc (bridgeLib, initFuncName);
- initFunc = *(msgPayload_createImpl*) &vp;
+ snprintf (initFuncName, 256, "%sPayload_createImpl", payloadName);
+ vp = loadLibFunc (bridgeLib, initFuncName);
+ initFunc = *(msgPayload_createImpl*) &vp;

if (!initFunc)
{
@@ -1887,7 +1926,7 @@ mama_loadPayloadBridgeInternal (mamaPayloadBridge*
impl,

if (MAMA_STATUS_OK != (status = initFunc (impl, &payloadChar)))
{
- wthread_static_mutex_unlock (&gImpl.myLock);
+ wthread_static_mutex_unlock (&gImpl.myLock);

return status;
}
@@ -1910,13 +1949,19 @@ mama_loadPayloadBridgeInternal
(mamaPayloadBridge* impl,
"Payload bridge %s already loaded",
payloadName);

- wthread_static_mutex_unlock (&gImpl.myLock);
+ wthread_static_mutex_unlock (&gImpl.myLock);

return MAMA_STATUS_OK;
}

+ /* Gives a warning - casting from void* to bridge_destroyImpl func */
+ snprintf (destroyFuncName, 256, "%sPayload_destroyImpl",
payloadName);
+ vp = loadLibFunc (bridgeLib, destroyFuncName);
+ destroyFunc = *(msgPayload_destroyImpl*) &vp;
+
gImpl.myPayloads [(int)payloadChar] = *impl;
gImpl.myPayloadLibraries [(int)payloadChar] = bridgeLib;
+ gImpl.myPayloadDestroyFuncs [(int)payloadChar] = destroyFunc;

if (!gDefaultPayload)
{
diff --git a/mama/c_cpp/src/c/mama/mama.h b/mama/c_cpp/src/c/mama/mama.h
index a574f36..07a9115 100644
--- a/mama/c_cpp/src/c/mama/mama.h
+++ b/mama/c_cpp/src/c/mama/mama.h
@@ -133,11 +133,34 @@ extern "C"
*/
MAMAExpDLL
extern mama_status
- mama_loadBridge (mamaBridge* impl, const char* middleware);
+ mama_loadBridge (mamaBridge* impl,
+ const char* middleware);

+ /**
+ * Load a payload bridge by its well-known library name string.
E.g. "wmsg"
+ * will load the WombatMsg payload, provided it exists on the system.
+ *
+ * @param bridge The payload bridge object.
+ * @param payloadName The well-known name of the payload library.
+ */
+ MAMAExpDLL
+ extern mama_status
+ mama_loadPayloadBridge (mamaPayloadBridge* bridge,
+ const char* payloadName);
+
+ /**
+ * Load a payload bridge by using the mamaPayloadType identifier.
This
+ * utilises mamaPayload_convertToLibString() in order to retrieve the
+ * well-known library string for the payload library, and this is
then
+ * delegated to mama_loadPayloadBridge.
+ *
+ * @param bridge The payload bridge object.
+ * @param payload The payload enumeration identifier.
+ */
MAMAExpDLL
extern mama_status
- mama_loadPayloadBridge (mamaPayloadBridge* bridge, const char*
payloadName);
+ mama_loadPayloadBridgeByType (mamaPayloadBridge* bridge,
+ mamaPayloadType payload);

/** Load the bridge specified by middleware string using the path
specified by the user.
* If the bridge has already been loaded then the existing bridge
instance
diff --git a/mama/c_cpp/src/c/mama/msg.h b/mama/c_cpp/src/c/mama/msg.h
index bf0d562..a9c3438 100644
--- a/mama/c_cpp/src/c/mama/msg.h
+++ b/mama/c_cpp/src/c/mama/msg.h
@@ -74,6 +74,36 @@ extern const char*
mamaPayload_convertToString (mamaPayloadType payloadType);

/**
+ * Convert a string to a mamaPayloadType.
+ *
+ * @param str The str to convert.
+ */
+MAMAExpDLL
+extern mamaPayloadType
+mamaPayload_convertFromString (const char* str);
+
+/**
+ * Convert a mamaPayloadType value to a well-known library name string
+ * for that payload. E.g. MAMA_PAYLOAD_WOMBAT_MSG translates into
+ * "wmsg" (of which the payload library would be libwmsgimpl.so on *nix).
+ *
+ * @param payloadType The payloadType to convert to a library name
string.
+ */
+MAMAExpDLL
+extern const char*
+mamaPayload_convertToLibString (mamaPayloadType payloadType);
+
+/**
+ * Convert a well-known library name to a mamaPayloadType value
+ * for that payload. E.g. "wmsg" translates into
MAMA_PAYLOAD_WOMBAT_MSG.
+ *
+ * @param str The library name string to convert to a mamaPayloadType.
+ */
+MAMAExpDLL
+extern mamaPayloadType
+mamaPayload_convertFromLibString (const char* str);
+
+/**
* Create a mamaMsg.
*
* @param msg The location of a mamaMsg where to store the result.
diff --git a/mama/c_cpp/src/c/msg.c b/mama/c_cpp/src/c/msg.c
index 286fcad..506c390 100644
--- a/mama/c_cpp/src/c/msg.c
+++ b/mama/c_cpp/src/c/msg.c
@@ -41,13 +41,39 @@

#include "wombat/wincompat.h"

-
#define MAX_SUBJECT 256

-#define NOFID 0
-
-static const int INCLUDE_FIELD_NAME = (1 << 4);
-static const int INCLUDE_FIELD_ID = (1 << 2);
+typedef struct PayloadTable
+{
+ mamaPayloadType mType;
+ const char* mName;
+ const char* mLibrary;
+}
+PayloadTable;
+
+static
+const PayloadTable kPayloads [] =
+{
+ {MAMA_PAYLOAD_ACTIV, "activ", "activmsg"},
+ {MAMA_PAYLOAD_AVIS, "AVIS", "avismsg"},
+ {MAMA_PAYLOAD_EXEGY, "EXEGY", "exegymsg"},
+ {MAMA_PAYLOAD_FAST, "FAST", "fastmsg"},
+ {MAMA_PAYLOAD_IBMWFO, "ibmwfo", "ibmwfomsg"},
+ {MAMA_PAYLOAD_INRUSH, "INRUSH", "inrushmsg"},
+ {MAMA_PAYLOAD_QPID, "QPID", "qpidmsg"},
+ {MAMA_PAYLOAD_RAI, "rai", "raimsg"},
+ {MAMA_PAYLOAD_SOLACE, "solacemsg", "solacemsg"},
+ {MAMA_PAYLOAD_TIBRV, "TIBRV", "tibrvmsg"},
+ {MAMA_PAYLOAD_TICK42BLP, "TICK42BLP", "tick42blpmsg"},
+ {MAMA_PAYLOAD_TICK42RMDS, "TICK42RMDS", "tick42rmdsmsg"},
+ {MAMA_PAYLOAD_UMS, "UMS", "umsmsg"},
+ {MAMA_PAYLOAD_V5, "V5", "wcachemsg"},
+ {MAMA_PAYLOAD_VULCAN, "Vulcan", "vulcanmsg"},
+ {MAMA_PAYLOAD_WOMBAT_MSG, "WombatMsg", "wmsg"},
+
+ /* Linear search requires unknown to be at the end-of-table */
+ {MAMA_PAYLOAD_UNKNOWN, "unknown", NULL}
+};

typedef struct mamaMsgIteratorImpl
{
@@ -106,6 +132,66 @@ typedef struct mamaMsgImpl_
static mama_status
mamaMsgImpl_destroyLastVectorMsg (mamaMsgImpl *impl);

+static
+const PayloadTable*
+mamaPayloadImpl_lookupTableByType (mamaPayloadType payloadType)
+{
+ int k = 0;
+ for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
+ {
+ if (kPayloads[k].mType == payloadType)
+ {
+ break;
+ }
+ }
+
+ return &kPayloads[k];
+}
+
+static
+const PayloadTable*
+mamaPayloadImpl_lookupTableByName (const char* name)
+{
+ int k = 0;
+
+ if (!name)
+ {
+ return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
+ }
+
+ for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
+ {
+ if (0 == strcasecmp (name, kPayloads[k].mName))
+ {
+ break;
+ }
+ }
+
+ return &kPayloads[k];
+}
+
+static
+const PayloadTable*
+mamaPayloadImpl_lookupTableByLibrary (const char* name)
+{
+ int k = 0;
+
+ if (!name)
+ {
+ return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
+ }
+
+ for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
+ {
+ if (0 == strcasecmp (name, kPayloads[k].mLibrary))
+ {
+ break;
+ }
+ }
+
+ return &kPayloads[k];
+}
+
/*=================================================================
= Public functions - defined in mama/msg.h
================================================================*/
@@ -166,7 +252,9 @@ mamaMsg_destroy (mamaMsg msg)
if (impl->mCurrentField)
{
mamaMsgField_destroy (impl->mCurrentField);
+ impl->mCurrentField = NULL;
}
+
/*Destroy the reusable field object*/
if (impl->mCopy)
{
@@ -366,44 +454,29 @@ mamaMsgImpl_getPayload (const mamaMsg msg,
msgPayload* payload)
const char*
mamaPayload_convertToString (mamaPayloadType payloadType)
{
- switch (payloadType)
- {
- case MAMA_PAYLOAD_SOLACE:
- return "solacemsg";
- case MAMA_PAYLOAD_V5:
- return "V5";
- case MAMA_PAYLOAD_AVIS:
- return "AVIS";
- case MAMA_PAYLOAD_TICK42BLP:
- return "TICK42BLP";
- case MAMA_PAYLOAD_FAST:
- return "FAST";
- case MAMA_PAYLOAD_RAI:
- return "rai";
- case MAMA_PAYLOAD_UMS:
- return "UMS";
- case MAMA_PAYLOAD_TICK42RMDS:
- return "TICK42RMDS";
- case MAMA_PAYLOAD_QPID:
- return "QPID";
- case MAMA_PAYLOAD_TIBRV:
- return "TIBRV";
- case MAMA_PAYLOAD_IBMWFO:
- return "ibmwfo";
- case MAMA_PAYLOAD_ACTIV:
- return "activ";
- case MAMA_PAYLOAD_VULCAN:
- return "Vulcan";
- case MAMA_PAYLOAD_WOMBAT_MSG:
- return "WombatMsg";
- case MAMA_PAYLOAD_EXEGY:
- return "EXEGY";
- case MAMA_PAYLOAD_INRUSH:
- return "INRUSH";
-
- default:
- return "unknown";
- }
+ const PayloadTable* table = mamaPayloadImpl_lookupTableByType
(payloadType);
+ return table->mName;
+}
+
+mamaPayloadType
+mamaPayload_convertFromString (const char* str)
+{
+ const PayloadTable* table = mamaPayloadImpl_lookupTableByName (str);
+ return table->mType;
+}
+
+const char*
+mamaPayload_convertToLibString (mamaPayloadType payloadType)
+{
+ const PayloadTable* table = mamaPayloadImpl_lookupTableByType
(payloadType);
+ return table->mLibrary;
+}
+
+mamaPayloadType
+mamaPayload_convertFromLibString (const char* str)
+{
+ const PayloadTable* table = mamaPayloadImpl_lookupTableByLibrary
(str);
+ return table->mType;
}

mama_status
@@ -3481,7 +3554,11 @@ mama_status mamaMsgIterator_destroy
(mamaMsgIterator iterator)
}

if (impl->mCurrentField)
+ {
mamaMsgField_destroy (impl->mCurrentField);
+ impl->mCurrentField = NULL;
+ }
+
free (impl);
return MAMA_STATUS_OK;
}
diff --git a/mama/c_cpp/src/c/payloadbridge.h
b/mama/c_cpp/src/c/payloadbridge.h
index 703c1a9..6ad7108 100644
--- a/mama/c_cpp/src/c/payloadbridge.h
+++ b/mama/c_cpp/src/c/payloadbridge.h
@@ -422,8 +422,12 @@ while(0)
\
typedef mama_status
(*msgPayload_createImpl) (mamaPayloadBridge* result, char*
identifier);

+typedef mama_status
+(*msgPayload_destroyImpl) (mamaPayloadBridge result);
+
typedef mamaPayloadType
(*msgPayload_getType) (void);
+
/*===================================================================
= msgPayload bridge function pointers =
====================================================================*/
diff --git a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
index ed18a64..1e14227 100644
--- a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
+++ b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
@@ -20,14 +20,9 @@ protected:

virtual void TearDown(void);

- typedef std::pair <mamaPayloadType, std::string> MamaPayloadPair;
- typedef std::map <mamaPayloadType, std::string> MamaPayloadMapType;
-
- typedef std::pair <mamaMiddleware, std::string> MamaMiddlewarePair;
- typedef std::map <mamaMiddleware, std::string>
MamaMiddlewareMapType;
-
- MamaPayloadMapType payloadTestData;
- MamaMiddlewareMapType middlewareTestData;
+ typedef std::pair <std::string, std::string> PayloadNameLib;
+ typedef std::map <mamaPayloadType, PayloadNameLib>
MamaPayloadMapType;
+ typedef std::map <mamaMiddleware, std::string>
MamaMiddlewareMapType;

private:
void CreateTestData();
@@ -55,100 +50,81 @@ MamaEnumTestsC::TearDown()
void
MamaEnumTestsC::CreateTestData()
{
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_V5,
"V5"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_AVIS,
"AVIS"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42BLP,
"TICK42BLP"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_FAST,
"FAST"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_RAI,
"rai"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UMS,
"UMS"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42RMDS,
"TICK42RMDS"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_QPID,
"QPID"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TIBRV,
"TIBRV"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_IBMWFO,
"ibmwfo"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_ACTIV,
"activ"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_VULCAN,
"Vulcan"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_WOMBAT_MSG,
"WombatMsg"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_EXEGY,
"EXEGY"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_SOLACE,
"solacemsg"));
- payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UNKNOWN,
"unknown"));
-
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_WMW,
"wmw"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_LBM,
"lbm"));
- middlewareTestData.insert (MamaMiddlewarePair
(MAMA_MIDDLEWARE_TIBRV, "tibrv"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_AVIS,
"AVIS"));
- middlewareTestData.insert (MamaMiddlewarePair
(MAMA_MIDDLEWARE_TICK42BLP, "tick42blp"));
- middlewareTestData.insert (MamaMiddlewarePair
(MAMA_MIDDLEWARE_SOLACE, "SOLACE"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_RAI,
"rai"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_QPID,
"QPID"));
- middlewareTestData.insert (MamaMiddlewarePair
(MAMA_MIDDLEWARE_EXEGY, "exegy"));
- middlewareTestData.insert (MamaMiddlewarePair
(MAMA_MIDDLEWARE_IBMWFO, "ibmwfo"));
- middlewareTestData.insert (MamaMiddlewarePair
(MAMA_MIDDLEWARE_ACTIV, "activ"));
- middlewareTestData.insert (MamaMiddlewarePair
(MAMA_MIDDLEWARE_TICK42RMDS, "tick42rmds"));
- middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_UMS,
"ums"));
- middlewareTestData.insert (MamaMiddlewarePair
(MAMA_MIDDLEWARE_VULCAN, "vulcan"));
- middlewareTestData.insert (MamaMiddlewarePair
(MAMA_MIDDLEWARE_UNKNOWN, "unknown"));
}

-
-TEST_F (MamaEnumTestsC, testPayloadConvertToString)
+TEST_F (MamaEnumTestsC, testPayloadStringConversions)
{
+ MamaPayloadMapType testData;
+ testData[MAMA_PAYLOAD_ACTIV] = PayloadNameLib ("activ",
"activmsg");
+ testData[MAMA_PAYLOAD_AVIS] = PayloadNameLib ("AVIS",
"avismsg");
+ testData[MAMA_PAYLOAD_EXEGY] = PayloadNameLib ("EXEGY",
"exegymsg");
+ testData[MAMA_PAYLOAD_FAST] = PayloadNameLib ("FAST",
"fastmsg");
+ testData[MAMA_PAYLOAD_IBMWFO] = PayloadNameLib ("ibmwfo",
"ibmwfomsg");
+ testData[MAMA_PAYLOAD_QPID] = PayloadNameLib ("QPID",
"qpidmsg");
+ testData[MAMA_PAYLOAD_RAI] = PayloadNameLib ("rai",
"raimsg");
+ testData[MAMA_PAYLOAD_SOLACE] = PayloadNameLib ("solacemsg",
"solacemsg");
+ testData[MAMA_PAYLOAD_TIBRV] = PayloadNameLib ("TIBRV",
"tibrvmsg");
+ testData[MAMA_PAYLOAD_TICK42BLP] = PayloadNameLib ("TICK42BLP",
"tick42blpmsg");
+ testData[MAMA_PAYLOAD_TICK42RMDS] = PayloadNameLib ("TICK42RMDS",
"tick42rmdsmsg");
+ testData[MAMA_PAYLOAD_UMS] = PayloadNameLib ("UMS",
"umsmsg");
+ testData[MAMA_PAYLOAD_VULCAN] = PayloadNameLib ("Vulcan",
"vulcanmsg");
+ testData[MAMA_PAYLOAD_WOMBAT_MSG] = PayloadNameLib ("WombatMsg",
"wmsg");
+ testData[MAMA_PAYLOAD_V5] = PayloadNameLib ("V5",
"wcachemsg");
+ testData[MAMA_PAYLOAD_UNKNOWN] = PayloadNameLib ("unknown",
"");
+
MamaPayloadMapType::iterator itr;
- bool passed = true;

- for(itr = payloadTestData.begin(); itr != payloadTestData.end();
++itr) {
+ for(itr = testData.begin(); itr != testData.end(); ++itr) {

- mamaPayloadType payload = (*itr).first;
- std::string expected = (*itr).second;
-
- std::string actual = mamaPayload_convertToString (payload);
-
- EXPECT_STREQ(actual.c_str(), expected.c_str());
-
- if (actual != expected)
- passed = false;
+ mamaPayloadType payload = itr->first;
+ PayloadNameLib& namelib = itr->second;
+ const char* name = namelib.first.c_str();
+ const char* library = namelib.second.c_str();
+
+ // Override for unknown, mama returns NULL for unknown library
(so that
+ // it prevents libraries from trying to load libmamaunknownmsg).
+ if (MAMA_PAYLOAD_UNKNOWN == payload)
+ {
+ library = NULL;
+ }
+
+ // Check name/payload conversion
+ EXPECT_STREQ(name, mamaPayload_convertToString (payload));
+ EXPECT_EQ((char)payload, (char)mamaPayload_convertFromString
(name));
+
+ // Check library/payload conversion
+ EXPECT_STREQ(library, mamaPayload_convertToLibString (payload));
+ EXPECT_EQ((char)payload, (char)mamaPayload_convertFromLibString
(library));
}
-
- ASSERT_EQ( passed, true );
}

-TEST_F (MamaEnumTestsC, testMiddlewareConvertToString)
+TEST_F (MamaEnumTestsC, testMiddlewareStringConversions)
{
+ MamaMiddlewareMapType testData;
+ testData[MAMA_MIDDLEWARE_WMW] = "wmw";
+ testData[MAMA_MIDDLEWARE_LBM] = "lbm";
+ testData[MAMA_MIDDLEWARE_TIBRV] = "tibrv";
+ testData[MAMA_MIDDLEWARE_AVIS] = "AVIS";
+ testData[MAMA_MIDDLEWARE_TICK42BLP] = "tick42blp";
+ testData[MAMA_MIDDLEWARE_SOLACE] = "SOLACE";
+ testData[MAMA_MIDDLEWARE_RAI] = "rai";
+ testData[MAMA_MIDDLEWARE_QPID] = "QPID";
+ testData[MAMA_MIDDLEWARE_EXEGY] = "exegy";
+ testData[MAMA_MIDDLEWARE_IBMWFO] = "ibmwfo";
+ testData[MAMA_MIDDLEWARE_ACTIV] = "activ";
+ testData[MAMA_MIDDLEWARE_TICK42RMDS] = "tick42rmds";
+ testData[MAMA_MIDDLEWARE_UMS] = "ums";
+ testData[MAMA_MIDDLEWARE_VULCAN] = "vulcan";
+ testData[MAMA_MIDDLEWARE_UNKNOWN] = "unknown";
MamaMiddlewareMapType::iterator itr;
- bool passed = true;

- for(itr = middlewareTestData.begin(); itr !=
middlewareTestData.end(); ++itr) {
+ for(itr = testData.begin(); itr != testData.end(); ++itr) {

- mamaMiddleware middleware = (*itr).first;
- std::string expected = (*itr).second;
-
- std::string actual = mamaMiddleware_convertToString (middleware);
+ mamaMiddleware middleware = itr->first;
+ const char* name = itr->second.c_str();

- EXPECT_STREQ(actual.c_str(), expected.c_str());
-
- if (actual != expected)
- passed = false;
+ EXPECT_STREQ(name, mamaMiddleware_convertToString (middleware));
+ EXPECT_EQ((char)middleware,
(char)mamaMiddleware_convertFromString (name));
}
-
- ASSERT_EQ (passed, true);
}

-TEST_F (MamaEnumTestsC, testMiddlewareConvertFromString)
-{
- MamaMiddlewareMapType::iterator itr;
- bool passed = true;
-
- for (itr = middlewareTestData.begin(); itr !=
middlewareTestData.end(); itr++) {
-
- mamaMiddleware expected = (*itr).first;
- std::string middleware = (*itr).second;
-
- mamaMiddleware actual = mamaMiddleware_convertFromString
(middleware.c_str());
-
- EXPECT_EQ (actual, expected);
-
- if (actual != expected)
- passed = false;
- }
-
- ASSERT_EQ (passed, true);
-}
________________________________________________________

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.
________________________________________________________


Lee Skillen <lskillen@...>
 

Hey Damian,

Thanks, we're happy to finally be in a better position to start contributing.   First of all, just so that focus isn't purely on the JIT loading, I wanted to point out that this patch also has fixes for memory leaks, a reworking of the payload conversion unit tests, and a new style for the payload mappings (which we hope will be useful from a maintenance perspective to keep things centralised).

The new unit tests exercise the the string conversion capability and we ran the entire suite to ensure that nothing else has been broken.  Memory leaks are confirmed for presence before patch via valgrind and memory sanitizer and their removal is confirmed after.  We also utilise helgrind and thread sanitizer for threading issues in general, although not for this patch.

The JIT loading mechanism has been tested locally using a modified version of avis and our middleware.   Setup for a simple test would be to load a consuming application (e.g. mamaconsumerc) with any middleware that is payload agnostic, and then send a message utilising a payload that is different than the default payload for that middleware.  Pre-patch this would crash the application, but post-patch it'll load the payload just-in-time.

For testing the receiving bit is fine but none of the testtools applications have the capability to specify a different payload other than the default. So for the sake of demonstration I modified mamaproducerc to be capable of specifying payload on the command-line via -P <payload>.  The test looks like this:

Pre-patch:

$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v
Segmentation fault (core dumped)

Post-patch:
 
$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
[snip]
2014-02-20 16:57:20: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]
 
$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]
Starting to publish

Looking back at the mamaconsumer log we can see that it successfully loaded the vulcanmsg payload JIT :-

2014-02-20 16:58:09: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
Rate: 2 Latency(usec): Low: 13775       Average: 447183 High: 910969    Gaps: 1
Rate: 9 Latency(usec): Low: 34754       Average: 507466 High: 933766    Gaps: 0
Rate: 9 Latency(usec): Low: 56764       Average: 528090 High: 954780    Gaps: 0

Hope that looks good for you guys - Please feel free to experiment on the new branch.  We'd really like to get this code in to the mainline if at all possible, and we've got quite a lot more code to contribute too.

Apologies about the carriage returns in the patch file - I don't personally use Windows and we utilise git-format-patch and git-send-email for the patch formatting and sending.  The patch prior to sending doesn't have any carriage return characters in it, but I looked at one I pulled back and it does have 0x0d 0x0a ("\r\n") for the line endings.  So it seems that either SMTP or git-send-email was the culprit.  I'll have a look into preventing it but in the short-term `git apply --whitespace=fix` should sort it. :-(

Cracking start for submitting patches, eh. :-)

Cheers,
Lee




On 20 February 2014 14:36, Damian Maguire <DMaguire@...> wrote:
Cheers for this Lee, looks very interesting. From a quick look/preliminary
test the changes look solid, but given the size and scope of the change I
think the best thing to do is fire it into it's own branch for a while and
let people check it out for themselves. As such I'm going to stick it into
a feature-jit-payload branch on the main repo. Anyone who's interested can
then have a look and comment on the changes, and we can fix any bugs there
as required.

Regarding testing these changes, have you guys actually got this running
with any of the open source bridges, or do you have any sample testing you
can show us to demonstrate that it both works, and doesn't break the
existing code?

Just a quick note on the patch files you've attached - they seem to have
landed (to me at least) with DOS line endings, which messes with the git
apply step. It's not a big deal, since I can easily enough strip the line
endings, but probably something to watch out for with future submissions
(particularly if you touch any of the unfortunate files that already mix
and match line endings).

I'm also taking a look at the other patches you've sent through, so I'll
get back to you on those.

Cheers,

Damian



On 2/19/14 3:53 PM, "Lee Skillen" <lskillen@...> wrote:

>
>If a library hasn't been previously loaded then OpenMAMA will now attempt
>to load it by converting the payload identifier into a well-known string
>for the payload library.  This behaviour is useful if the payload isn't
>known ahead of time or if lazy loading is desired over enumeration of a
>existing set of libraries.  No changes made to existing API interfaces.
>
>The following functions have been added :-
>
>- mamaPayload_convertToLibString():
>  Maps a mamaPayloadType enumeration to the well-known library name.
>
>- mamaPayload_convertFromLibString():
>  Inverse of mamaPayload_convertToLibString.
>
>- mama_loadPayloadBridgeByType():
>  Takes a mamaPayloadType and utilises mamaPayload_convertToLibString()
>  to retrieve the library name for the payload, and then delegates this
>  to the existing mama_loadPayloadBridge() function.
>
>The string conversions are implemented as a linear search, primarily due
>to these functions not being on the hot path - Their use is prior to a
>library being loaded and (I would imagine) would be seldom used in other
>situations.
>
>Additional changes :-
>
>If msgPayload_destroyImpl has been exposed by the payload library then
>this is now utilised to cleanup the payload upon shutdown properly.  It
>was observed that each payload always leaked memory due to lacking the
>capability to destroy itself.  Individual payloads will still need to
>expose this - Backwards compatible as it won't be called otherwise.
>
>The MamaEnumTestsC set of unit tests have been rewritten to utilise the
>new conversion functionality.  The tests have also been renamed as such:
>
>- testPayloadToString -> testPayloadStringConversions
>- testPayloadFromString -> testPayloadStringConversions
>- testMiddlewareToString -> testMiddlewareStringConversions
>- testMiddlewareFromString -> testMiddlewareStringConversions
>
>Noticed that the ASSERT_EQ/EXPECT_EQ statements in gtest are sometimes
>used incorrectly, in that the expected/actual are reversed (it should be
>expected on the left and actual on the right).
>
>Todo :-
>
>Need to confirm well-known library names with all vendors - Will only be
>important if JIT loading is actually used, but the names should be made
>accurate anyway.
>
>Signed-off-by: Lee Skillen <lskillen@...>
>---
> mama/c_cpp/src/c/mama.c                            |  77 ++++++++--
> mama/c_cpp/src/c/mama/mama.h                       |  27 +++-
> mama/c_cpp/src/c/mama/msg.h                        |  30 ++++
> mama/c_cpp/src/c/msg.c                             | 163
>+++++++++++++++------
> mama/c_cpp/src/c/payloadbridge.h                   |   4 +
> .../src/gunittest/c/payloadmiddlewareidtest.cpp    | 152
>++++++++-----------
> 6 files changed, 304 insertions(+), 149 deletions(-)
>
>diff --git a/mama/c_cpp/src/c/mama.c b/mama/c_cpp/src/c/mama.c
>index 33a0768..3a7d7be 100644
>--- a/mama/c_cpp/src/c/mama.c
>+++ b/mama/c_cpp/src/c/mama.c
>@@ -28,6 +28,7 @@
> #include "wombat/strutils.h"
>
> #include <mama/mama.h>
>+#include <mama/msg.h>
> #include <mama/error.h>
> #include <mamainternal.h>
> #include <mama/version.h>
>@@ -168,6 +169,7 @@ typedef struct mamaImpl_
>     mamaPayloadBridge      myPayloads[MAMA_PAYLOAD_MAX];
>     LIB_HANDLE             myBridgeLibraries[MAMA_MIDDLEWARE_MAX];
>     LIB_HANDLE             myPayloadLibraries[MAMA_PAYLOAD_MAX];
>+    msgPayload_destroyImpl myPayloadDestroyFuncs[MAMA_PAYLOAD_MAX];
>     unsigned int           myRefCount;
>     wthread_static_mutex_t myLock;
> } mamaImpl;
>@@ -175,7 +177,7 @@ typedef struct mamaImpl_
> static mamaApplicationContext  appContext;
> static char mama_ver_string[256];
>
>-static mamaImpl gImpl = {{0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>+static mamaImpl gImpl = {{0}, {0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>
> /*
>*************************************************************************
>*/
> /* Private Function Prototypes. */
>@@ -624,10 +626,21 @@ mamaInternal_findBridge ()
> mamaPayloadBridge
> mamaInternal_findPayload (char id)
> {
>+    mamaPayloadBridge bridge = NULL;
>+
>     if ('\0' == id)
>         return NULL;
>
>-    return gImpl.myPayloads[(uint8_t)id];
>+    bridge = gImpl.myPayloads[(uint8_t)id];
>+
>+    if (NULL == bridge)
>+    {
>+        /* Dynamically load the payload bridge just-in-time, if it
>hasn't
>+         * already been loaded previously. */
>+        mama_loadPayloadBridgeByType (&bridge, (mamaPayloadType)id);
>+    }
>+
>+    return bridge;
> }
>
> mamaPayloadBridge
>@@ -1241,17 +1254,33 @@ mama_closeCount (unsigned int* count)
>
>         cleanupReservedFields();
>
>-         /* Look for a bridge for each of the payloads and close them */
>+        /* Look for a bridge for each of the payloads and close them */
>         for (payload = 0; payload != MAMA_PAYLOAD_MAX; ++payload)
>         {
>-              /* mamaPayloadBridgeImpl* impl = (mamaPayloadBridgeImpl*)
>-             * gImpl.myPayloads [(uint8_t)payload];*/
>-            gImpl.myPayloads[(uint8_t)payload] = NULL;
>-            if(gImpl.myPayloadLibraries[(uint8_t)payload])
>+            uint8_t                id   = (uint8_t)payload;
>+            LIB_HANDLE             lib  = gImpl.myPayloadLibraries[id];
>+            mamaPayloadBridgeImpl* impl = gImpl.myPayloads[id];
>+
>+            if (!lib)
>             {
>-                closeSharedLib
>(gImpl.myPayloadLibraries[(uint8_t)payload]);
>-                gImpl.myPayloadLibraries[(uint8_t)payload] = NULL;
>+                continue;
>+            }
>+
>+            if (impl)
>+            {
>+                msgPayload_destroyImpl destroyFunc =
>+                    gImpl.myPayloadDestroyFuncs[id];
>+
>+                if (destroyFunc)
>+                {
>+                    destroyFunc (impl);
>+                }
>+
>+                gImpl.myPayloads[id] = NULL;
>             }
>+
>+            closeSharedLib (lib);
>+            gImpl.myPayloadLibraries[id] = NULL;
>         }
>
>        gDefaultPayload = NULL;
>@@ -1827,18 +1856,29 @@ mama_setDefaultPayload (char id)
>
> mama_status
> mama_loadPayloadBridge (mamaPayloadBridge* impl,
>-                         const char*        payloadName)
>+                        const char*        payloadName)
> {
>     return mama_loadPayloadBridgeInternal (impl, payloadName);
> }
>+
>+mama_status
>+mama_loadPayloadBridgeByType (mamaPayloadBridge* impl,
>+                              mamaPayloadType    payload)
>+{
>+    const char* name = mamaPayload_convertToLibString (payload);
>+    return mama_loadPayloadBridgeInternal (impl, name);
>+}
>+
> mama_status
> mama_loadPayloadBridgeInternal  (mamaPayloadBridge* impl,
>                                  const char*        payloadName)
> {
>     char                    bridgeImplName  [256];
>     char                    initFuncName    [256];
>+    char                    destroyFuncName [256];
>     LIB_HANDLE              bridgeLib       = NULL;
>     msgPayload_createImpl   initFunc        = NULL;
>+    msgPayload_destroyImpl  destroyFunc     = NULL;
>     mama_status             status          = MAMA_STATUS_OK;
>     char                    payloadChar       ='\0';
>       void*                                   vp                              = NULL;
>@@ -1865,11 +1905,10 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>         return MAMA_STATUS_NO_BRIDGE_IMPL;
>     }
>
>-    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>-
>     /* Gives a warning - casting from void* to bridge_createImpl func */
>-      vp = loadLibFunc (bridgeLib, initFuncName);
>-      initFunc  = *(msgPayload_createImpl*) &vp;
>+    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>+    vp = loadLibFunc (bridgeLib, initFuncName);
>+    initFunc  = *(msgPayload_createImpl*) &vp;
>
>     if (!initFunc)
>     {
>@@ -1887,7 +1926,7 @@ mama_loadPayloadBridgeInternal  (mamaPayloadBridge*
>impl,
>
>     if (MAMA_STATUS_OK != (status = initFunc (impl, &payloadChar)))
>     {
>-       wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return status;
>     }
>@@ -1910,13 +1949,19 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>              "Payload bridge %s already loaded",
>              payloadName);
>
>-         wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return MAMA_STATUS_OK;
>     }
>
>+    /* Gives a warning - casting from void* to bridge_destroyImpl func */
>+    snprintf (destroyFuncName, 256, "%sPayload_destroyImpl",
>payloadName);
>+    vp = loadLibFunc (bridgeLib, destroyFuncName);
>+    destroyFunc = *(msgPayload_destroyImpl*) &vp;
>+
>     gImpl.myPayloads [(int)payloadChar] = *impl;
>     gImpl.myPayloadLibraries [(int)payloadChar] = bridgeLib;
>+    gImpl.myPayloadDestroyFuncs [(int)payloadChar] = destroyFunc;
>
>     if (!gDefaultPayload)
>     {
>diff --git a/mama/c_cpp/src/c/mama/mama.h b/mama/c_cpp/src/c/mama/mama.h
>index a574f36..07a9115 100644
>--- a/mama/c_cpp/src/c/mama/mama.h
>+++ b/mama/c_cpp/src/c/mama/mama.h
>@@ -133,11 +133,34 @@ extern "C"
>      */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadBridge (mamaBridge* impl, const char*  middleware);
>+    mama_loadBridge (mamaBridge* impl,
>+                     const char* middleware);
>
>+    /**
>+     * Load a payload bridge by its well-known library name string.
>E.g. "wmsg"
>+     * will load the WombatMsg payload, provided it exists on the system.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payloadName The well-known name of the payload library.
>+     */
>+    MAMAExpDLL
>+    extern mama_status
>+    mama_loadPayloadBridge (mamaPayloadBridge* bridge,
>+                            const char*        payloadName);
>+
>+    /**
>+     * Load a payload bridge by using the mamaPayloadType identifier.
>This
>+     * utilises mamaPayload_convertToLibString() in order to retrieve the
>+     * well-known library string for the payload library, and this is
>then
>+     * delegated to mama_loadPayloadBridge.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payload The payload enumeration identifier.
>+     */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadPayloadBridge (mamaPayloadBridge*  bridge,  const char*
>payloadName);
>+    mama_loadPayloadBridgeByType (mamaPayloadBridge* bridge,
>+                                  mamaPayloadType    payload);
>
>     /** Load the bridge specified by middleware string using the path
>specified by the user.
>      * If the bridge has already been loaded then the existing bridge
>instance
>diff --git a/mama/c_cpp/src/c/mama/msg.h b/mama/c_cpp/src/c/mama/msg.h
>index bf0d562..a9c3438 100644
>--- a/mama/c_cpp/src/c/mama/msg.h
>+++ b/mama/c_cpp/src/c/mama/msg.h
>@@ -74,6 +74,36 @@ extern const char*
> mamaPayload_convertToString (mamaPayloadType payloadType);
>
> /**
>+ * Convert a string to a mamaPayloadType.
>+ *
>+ * @param str The str to convert.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromString (const char* str);
>+
>+/**
>+ * Convert a mamaPayloadType value to a well-known library name string
>+ * for that payload.  E.g. MAMA_PAYLOAD_WOMBAT_MSG translates into
>+ * "wmsg" (of which the payload library would be libwmsgimpl.so on *nix).
>+ *
>+ * @param payloadType The payloadType to convert to a library name
>string.
>+ */
>+MAMAExpDLL
>+extern const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType);
>+
>+/**
>+ * Convert a well-known library name to a mamaPayloadType value
>+ * for that payload.  E.g. "wmsg" translates into
>MAMA_PAYLOAD_WOMBAT_MSG.
>+ *
>+ * @param str The library name string to convert to a mamaPayloadType.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str);
>+
>+/**
>  * Create a mamaMsg.
>  *
>  * @param msg The location of a mamaMsg where to store the result.
>diff --git a/mama/c_cpp/src/c/msg.c b/mama/c_cpp/src/c/msg.c
>index 286fcad..506c390 100644
>--- a/mama/c_cpp/src/c/msg.c
>+++ b/mama/c_cpp/src/c/msg.c
>@@ -41,13 +41,39 @@
>
> #include "wombat/wincompat.h"
>
>-
> #define MAX_SUBJECT 256
>
>-#define NOFID 0
>-
>-static const int INCLUDE_FIELD_NAME = (1 << 4);
>-static const int INCLUDE_FIELD_ID   = (1 << 2);
>+typedef struct PayloadTable
>+{
>+    mamaPayloadType mType;
>+    const char*     mName;
>+    const char*     mLibrary;
>+}
>+PayloadTable;
>+
>+static
>+const PayloadTable kPayloads [] =
>+{
>+    {MAMA_PAYLOAD_ACTIV,      "activ",      "activmsg"},
>+    {MAMA_PAYLOAD_AVIS,       "AVIS",       "avismsg"},
>+    {MAMA_PAYLOAD_EXEGY,      "EXEGY",      "exegymsg"},
>+    {MAMA_PAYLOAD_FAST,       "FAST",       "fastmsg"},
>+    {MAMA_PAYLOAD_IBMWFO,     "ibmwfo",     "ibmwfomsg"},
>+    {MAMA_PAYLOAD_INRUSH,     "INRUSH",     "inrushmsg"},
>+    {MAMA_PAYLOAD_QPID,       "QPID",       "qpidmsg"},
>+    {MAMA_PAYLOAD_RAI,        "rai",        "raimsg"},
>+    {MAMA_PAYLOAD_SOLACE,     "solacemsg",  "solacemsg"},
>+    {MAMA_PAYLOAD_TIBRV,      "TIBRV",      "tibrvmsg"},
>+    {MAMA_PAYLOAD_TICK42BLP,  "TICK42BLP",  "tick42blpmsg"},
>+    {MAMA_PAYLOAD_TICK42RMDS, "TICK42RMDS", "tick42rmdsmsg"},
>+    {MAMA_PAYLOAD_UMS,        "UMS",        "umsmsg"},
>+    {MAMA_PAYLOAD_V5,         "V5",         "wcachemsg"},
>+    {MAMA_PAYLOAD_VULCAN,     "Vulcan",     "vulcanmsg"},
>+    {MAMA_PAYLOAD_WOMBAT_MSG, "WombatMsg",  "wmsg"},
>+
>+    /* Linear search requires unknown to be at the end-of-table */
>+    {MAMA_PAYLOAD_UNKNOWN,    "unknown",    NULL}
>+};
>
> typedef struct mamaMsgIteratorImpl
> {
>@@ -106,6 +132,66 @@ typedef struct mamaMsgImpl_
> static mama_status
> mamaMsgImpl_destroyLastVectorMsg (mamaMsgImpl *impl);
>
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByType (mamaPayloadType payloadType)
>+{
>+    int k = 0;
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (kPayloads[k].mType == payloadType)
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByName (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mName))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByLibrary (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mLibrary))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
> /*=================================================================
>   = Public functions - defined in mama/msg.h
>   ================================================================*/
>@@ -166,7 +252,9 @@ mamaMsg_destroy (mamaMsg msg)
>     if (impl->mCurrentField)
>     {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>     }
>+
>     /*Destroy the reusable field object*/
>     if (impl->mCopy)
>     {
>@@ -366,44 +454,29 @@ mamaMsgImpl_getPayload (const mamaMsg msg,
>msgPayload* payload)
> const char*
> mamaPayload_convertToString (mamaPayloadType payloadType)
> {
>-    switch (payloadType)
>-    {
>-        case MAMA_PAYLOAD_SOLACE:
>-            return "solacemsg";
>-        case MAMA_PAYLOAD_V5:
>-            return "V5";
>-        case MAMA_PAYLOAD_AVIS:
>-            return "AVIS";
>-        case MAMA_PAYLOAD_TICK42BLP:
>-            return "TICK42BLP";
>-        case MAMA_PAYLOAD_FAST:
>-            return "FAST";
>-        case MAMA_PAYLOAD_RAI:
>-            return "rai";
>-        case MAMA_PAYLOAD_UMS:
>-            return "UMS";
>-        case MAMA_PAYLOAD_TICK42RMDS:
>-            return "TICK42RMDS";
>-        case MAMA_PAYLOAD_QPID:
>-            return "QPID";
>-        case MAMA_PAYLOAD_TIBRV:
>-            return "TIBRV";
>-        case MAMA_PAYLOAD_IBMWFO:
>-            return "ibmwfo";
>-        case MAMA_PAYLOAD_ACTIV:
>-            return "activ";
>-        case MAMA_PAYLOAD_VULCAN:
>-            return "Vulcan";
>-        case MAMA_PAYLOAD_WOMBAT_MSG:
>-            return "WombatMsg";
>-        case MAMA_PAYLOAD_EXEGY:
>-            return "EXEGY";
>-        case MAMA_PAYLOAD_INRUSH:
>-            return "INRUSH";
>-
>-        default:
>-            return "unknown";
>-    }
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mName;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByName (str);
>+    return table->mType;
>+}
>+
>+const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mLibrary;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByLibrary
>(str);
>+    return table->mType;
> }
>
> mama_status
>@@ -3481,7 +3554,11 @@ mama_status mamaMsgIterator_destroy
>(mamaMsgIterator iterator)
>     }
>
>     if (impl->mCurrentField)
>+    {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>+    }
>+
>     free (impl);
>     return MAMA_STATUS_OK;
> }
>diff --git a/mama/c_cpp/src/c/payloadbridge.h
>b/mama/c_cpp/src/c/payloadbridge.h
>index 703c1a9..6ad7108 100644
>--- a/mama/c_cpp/src/c/payloadbridge.h
>+++ b/mama/c_cpp/src/c/payloadbridge.h
>@@ -422,8 +422,12 @@ while(0)
>                       \
> typedef mama_status
> (*msgPayload_createImpl)      (mamaPayloadBridge* result, char*
>identifier);
>
>+typedef mama_status
>+(*msgPayload_destroyImpl)     (mamaPayloadBridge result);
>+
> typedef mamaPayloadType
> (*msgPayload_getType)         (void);
>+
> /*===================================================================
>  =              msgPayload bridge function pointers                 =
>  ====================================================================*/
>diff --git a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>index ed18a64..1e14227 100644
>--- a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>+++ b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>@@ -20,14 +20,9 @@ protected:
>
>    virtual void TearDown(void);
>
>-   typedef std::pair <mamaPayloadType, std::string> MamaPayloadPair;
>-   typedef std::map  <mamaPayloadType, std::string> MamaPayloadMapType;
>-
>-   typedef std::pair <mamaMiddleware,  std::string> MamaMiddlewarePair;
>-   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>-
>-   MamaPayloadMapType     payloadTestData;
>-   MamaMiddlewareMapType  middlewareTestData;
>+   typedef std::pair <std::string,     std::string>    PayloadNameLib;
>+   typedef std::map  <mamaPayloadType, PayloadNameLib>
>MamaPayloadMapType;
>+   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>
> private:
>    void CreateTestData();
>@@ -55,100 +50,81 @@ MamaEnumTestsC::TearDown()
> void
> MamaEnumTestsC::CreateTestData()
> {
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_V5,
>"V5"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_AVIS,
>"AVIS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42BLP,
>"TICK42BLP"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_FAST,
>"FAST"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_RAI,
>"rai"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UMS,
>"UMS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42RMDS,
>"TICK42RMDS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_QPID,
>"QPID"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TIBRV,
>"TIBRV"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_IBMWFO,
>"ibmwfo"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_ACTIV,
>"activ"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_VULCAN,
>"Vulcan"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_WOMBAT_MSG,
>"WombatMsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_EXEGY,
>"EXEGY"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_SOLACE,
>"solacemsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UNKNOWN,
>"unknown"));
>-
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_WMW,
>     "wmw"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_LBM,
>     "lbm"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TIBRV,     "tibrv"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_AVIS,
>     "AVIS"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42BLP, "tick42blp"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_SOLACE,    "SOLACE"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_RAI,
>     "rai"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_QPID,
>     "QPID"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_EXEGY,     "exegy"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_IBMWFO,    "ibmwfo"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_ACTIV,     "activ"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42RMDS, "tick42rmds"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_UMS,
>     "ums"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_VULCAN,    "vulcan"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_UNKNOWN,   "unknown"));
> }
>
>-
>-TEST_F (MamaEnumTestsC, testPayloadConvertToString)
>+TEST_F (MamaEnumTestsC, testPayloadStringConversions)
> {
>+    MamaPayloadMapType testData;
>+    testData[MAMA_PAYLOAD_ACTIV]      = PayloadNameLib ("activ",
>"activmsg");
>+    testData[MAMA_PAYLOAD_AVIS]       = PayloadNameLib ("AVIS",
>"avismsg");
>+    testData[MAMA_PAYLOAD_EXEGY]      = PayloadNameLib ("EXEGY",
>"exegymsg");
>+    testData[MAMA_PAYLOAD_FAST]       = PayloadNameLib ("FAST",
>"fastmsg");
>+    testData[MAMA_PAYLOAD_IBMWFO]     = PayloadNameLib ("ibmwfo",
>"ibmwfomsg");
>+    testData[MAMA_PAYLOAD_QPID]       = PayloadNameLib ("QPID",
>"qpidmsg");
>+    testData[MAMA_PAYLOAD_RAI]        = PayloadNameLib ("rai",
>"raimsg");
>+    testData[MAMA_PAYLOAD_SOLACE]     = PayloadNameLib ("solacemsg",
>"solacemsg");
>+    testData[MAMA_PAYLOAD_TIBRV]      = PayloadNameLib ("TIBRV",
>"tibrvmsg");
>+    testData[MAMA_PAYLOAD_TICK42BLP]  = PayloadNameLib ("TICK42BLP",
>"tick42blpmsg");
>+    testData[MAMA_PAYLOAD_TICK42RMDS] = PayloadNameLib ("TICK42RMDS",
>"tick42rmdsmsg");
>+    testData[MAMA_PAYLOAD_UMS]        = PayloadNameLib ("UMS",
>"umsmsg");
>+    testData[MAMA_PAYLOAD_VULCAN]     = PayloadNameLib ("Vulcan",
>"vulcanmsg");
>+    testData[MAMA_PAYLOAD_WOMBAT_MSG] = PayloadNameLib ("WombatMsg",
>"wmsg");
>+    testData[MAMA_PAYLOAD_V5]         = PayloadNameLib ("V5",
>"wcachemsg");
>+    testData[MAMA_PAYLOAD_UNKNOWN]    = PayloadNameLib ("unknown",
>"");
>+
>     MamaPayloadMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = payloadTestData.begin(); itr != payloadTestData.end();
>++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaPayloadType payload   = (*itr).first;
>-       std::string     expected  = (*itr).second;
>-
>-       std::string actual = mamaPayload_convertToString (payload);
>-
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       mamaPayloadType payload = itr->first;
>+       PayloadNameLib& namelib = itr->second;
>+       const char*     name    = namelib.first.c_str();
>+       const char*     library = namelib.second.c_str();
>+
>+       // Override for unknown, mama returns NULL for unknown library
>(so that
>+       // it prevents libraries from trying to load libmamaunknownmsg).
>+       if (MAMA_PAYLOAD_UNKNOWN == payload)
>+       {
>+           library = NULL;
>+       }
>+
>+       // Check name/payload conversion
>+       EXPECT_STREQ(name, mamaPayload_convertToString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromString
>(name));
>+
>+       // Check library/payload conversion
>+       EXPECT_STREQ(library, mamaPayload_convertToLibString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromLibString
>(library));
>     }
>-
>-    ASSERT_EQ( passed, true );
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertToString)
>+TEST_F (MamaEnumTestsC, testMiddlewareStringConversions)
> {
>+    MamaMiddlewareMapType testData;
>+    testData[MAMA_MIDDLEWARE_WMW]        = "wmw";
>+    testData[MAMA_MIDDLEWARE_LBM]        = "lbm";
>+    testData[MAMA_MIDDLEWARE_TIBRV]      = "tibrv";
>+    testData[MAMA_MIDDLEWARE_AVIS]       = "AVIS";
>+    testData[MAMA_MIDDLEWARE_TICK42BLP]  = "tick42blp";
>+    testData[MAMA_MIDDLEWARE_SOLACE]     = "SOLACE";
>+    testData[MAMA_MIDDLEWARE_RAI]        = "rai";
>+    testData[MAMA_MIDDLEWARE_QPID]       = "QPID";
>+    testData[MAMA_MIDDLEWARE_EXEGY]      = "exegy";
>+    testData[MAMA_MIDDLEWARE_IBMWFO]     = "ibmwfo";
>+    testData[MAMA_MIDDLEWARE_ACTIV]      = "activ";
>+    testData[MAMA_MIDDLEWARE_TICK42RMDS] = "tick42rmds";
>+    testData[MAMA_MIDDLEWARE_UMS]        = "ums";
>+    testData[MAMA_MIDDLEWARE_VULCAN]     = "vulcan";
>+    testData[MAMA_MIDDLEWARE_UNKNOWN]    = "unknown";
>     MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); ++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaMiddleware middleware = (*itr).first;
>-       std::string    expected   = (*itr).second;
>-
>-       std::string actual = mamaMiddleware_convertToString (middleware);
>+       mamaMiddleware middleware = itr->first;
>+       const char*    name       = itr->second.c_str();
>
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       EXPECT_STREQ(name, mamaMiddleware_convertToString (middleware));
>+       EXPECT_EQ((char)middleware,
>(char)mamaMiddleware_convertFromString (name));
>     }
>-
>-    ASSERT_EQ (passed, true);
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertFromString)
>-{
>-    MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>-
>-    for (itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); itr++) {
>-
>-        mamaMiddleware expected    = (*itr).first;
>-        std::string    middleware  = (*itr).second;
>-
>-        mamaMiddleware actual      = mamaMiddleware_convertFromString
>(middleware.c_str());
>-
>-        EXPECT_EQ (actual, expected);
>-
>-        if (actual != expected)
>-            passed = false;
>-    }
>-
>-    ASSERT_EQ (passed, true);
>-}
>

________________________________________________________

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.
________________________________________________________




--
Lee Skillen

Vulcan Financial Technologies
51 Malone Road, Belfast, BT9 6RY

Office:  +44 (0)28 95 817888
Mobile:  +44 (0)78 41 425152
Web:     www.vulcanft.com 


Lee Skillen <lskillen@...>
 

Hey,

Just a quick update, it looks like our SMTP server caused the carriage returns - I've modified a version of git-send-email to include the capability to mime encode attachments as base64.  Tested and it carries through the patch in verbatim, so I shouldn't be sending any more patches that look like they came from a Windows machine. ;-)  I'll tidy up the git patch and submit it back to git-core next week since it seems useful (and may be useful to others here too).  Let me know if you need to me to re-submit the OpenMAMA patches I've already sent.

Thanks,
Lee



On 20 February 2014 17:08, Lee Skillen <lskillen@...> wrote:
Hey Damian,

Thanks, we're happy to finally be in a better position to start contributing.   First of all, just so that focus isn't purely on the JIT loading, I wanted to point out that this patch also has fixes for memory leaks, a reworking of the payload conversion unit tests, and a new style for the payload mappings (which we hope will be useful from a maintenance perspective to keep things centralised).

The new unit tests exercise the the string conversion capability and we ran the entire suite to ensure that nothing else has been broken.  Memory leaks are confirmed for presence before patch via valgrind and memory sanitizer and their removal is confirmed after.  We also utilise helgrind and thread sanitizer for threading issues in general, although not for this patch.

The JIT loading mechanism has been tested locally using a modified version of avis and our middleware.   Setup for a simple test would be to load a consuming application (e.g. mamaconsumerc) with any middleware that is payload agnostic, and then send a message utilising a payload that is different than the default payload for that middleware.  Pre-patch this would crash the application, but post-patch it'll load the payload just-in-time.

For testing the receiving bit is fine but none of the testtools applications have the capability to specify a different payload other than the default. So for the sake of demonstration I modified mamaproducerc to be capable of specifying payload on the command-line via -P <payload>.  The test looks like this:

Pre-patch:

$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v
Segmentation fault (core dumped)

Post-patch:
 
$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
[snip]
2014-02-20 16:57:20: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]
 
$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]
Starting to publish

Looking back at the mamaconsumer log we can see that it successfully loaded the vulcanmsg payload JIT :-

2014-02-20 16:58:09: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
Rate: 2 Latency(usec): Low: 13775       Average: 447183 High: 910969    Gaps: 1
Rate: 9 Latency(usec): Low: 34754       Average: 507466 High: 933766    Gaps: 0
Rate: 9 Latency(usec): Low: 56764       Average: 528090 High: 954780    Gaps: 0

Hope that looks good for you guys - Please feel free to experiment on the new branch.  We'd really like to get this code in to the mainline if at all possible, and we've got quite a lot more code to contribute too.

Apologies about the carriage returns in the patch file - I don't personally use Windows and we utilise git-format-patch and git-send-email for the patch formatting and sending.  The patch prior to sending doesn't have any carriage return characters in it, but I looked at one I pulled back and it does have 0x0d 0x0a ("\r\n") for the line endings.  So it seems that either SMTP or git-send-email was the culprit.  I'll have a look into preventing it but in the short-term `git apply --whitespace=fix` should sort it. :-(

Cracking start for submitting patches, eh. :-)

Cheers,
Lee




On 20 February 2014 14:36, Damian Maguire <DMaguire@...> wrote:
Cheers for this Lee, looks very interesting. From a quick look/preliminary
test the changes look solid, but given the size and scope of the change I
think the best thing to do is fire it into it's own branch for a while and
let people check it out for themselves. As such I'm going to stick it into
a feature-jit-payload branch on the main repo. Anyone who's interested can
then have a look and comment on the changes, and we can fix any bugs there
as required.

Regarding testing these changes, have you guys actually got this running
with any of the open source bridges, or do you have any sample testing you
can show us to demonstrate that it both works, and doesn't break the
existing code?

Just a quick note on the patch files you've attached - they seem to have
landed (to me at least) with DOS line endings, which messes with the git
apply step. It's not a big deal, since I can easily enough strip the line
endings, but probably something to watch out for with future submissions
(particularly if you touch any of the unfortunate files that already mix
and match line endings).

I'm also taking a look at the other patches you've sent through, so I'll
get back to you on those.

Cheers,

Damian



On 2/19/14 3:53 PM, "Lee Skillen" <lskillen@...> wrote:

>
>If a library hasn't been previously loaded then OpenMAMA will now attempt
>to load it by converting the payload identifier into a well-known string
>for the payload library.  This behaviour is useful if the payload isn't
>known ahead of time or if lazy loading is desired over enumeration of a
>existing set of libraries.  No changes made to existing API interfaces.
>
>The following functions have been added :-
>
>- mamaPayload_convertToLibString():
>  Maps a mamaPayloadType enumeration to the well-known library name.
>
>- mamaPayload_convertFromLibString():
>  Inverse of mamaPayload_convertToLibString.
>
>- mama_loadPayloadBridgeByType():
>  Takes a mamaPayloadType and utilises mamaPayload_convertToLibString()
>  to retrieve the library name for the payload, and then delegates this
>  to the existing mama_loadPayloadBridge() function.
>
>The string conversions are implemented as a linear search, primarily due
>to these functions not being on the hot path - Their use is prior to a
>library being loaded and (I would imagine) would be seldom used in other
>situations.
>
>Additional changes :-
>
>If msgPayload_destroyImpl has been exposed by the payload library then
>this is now utilised to cleanup the payload upon shutdown properly.  It
>was observed that each payload always leaked memory due to lacking the
>capability to destroy itself.  Individual payloads will still need to
>expose this - Backwards compatible as it won't be called otherwise.
>
>The MamaEnumTestsC set of unit tests have been rewritten to utilise the
>new conversion functionality.  The tests have also been renamed as such:
>
>- testPayloadToString -> testPayloadStringConversions
>- testPayloadFromString -> testPayloadStringConversions
>- testMiddlewareToString -> testMiddlewareStringConversions
>- testMiddlewareFromString -> testMiddlewareStringConversions
>
>Noticed that the ASSERT_EQ/EXPECT_EQ statements in gtest are sometimes
>used incorrectly, in that the expected/actual are reversed (it should be
>expected on the left and actual on the right).
>
>Todo :-
>
>Need to confirm well-known library names with all vendors - Will only be
>important if JIT loading is actually used, but the names should be made
>accurate anyway.
>
>Signed-off-by: Lee Skillen <lskillen@...>
>---
> mama/c_cpp/src/c/mama.c                            |  77 ++++++++--
> mama/c_cpp/src/c/mama/mama.h                       |  27 +++-
> mama/c_cpp/src/c/mama/msg.h                        |  30 ++++
> mama/c_cpp/src/c/msg.c                             | 163
>+++++++++++++++------
> mama/c_cpp/src/c/payloadbridge.h                   |   4 +
> .../src/gunittest/c/payloadmiddlewareidtest.cpp    | 152
>++++++++-----------
> 6 files changed, 304 insertions(+), 149 deletions(-)
>
>diff --git a/mama/c_cpp/src/c/mama.c b/mama/c_cpp/src/c/mama.c
>index 33a0768..3a7d7be 100644
>--- a/mama/c_cpp/src/c/mama.c
>+++ b/mama/c_cpp/src/c/mama.c
>@@ -28,6 +28,7 @@
> #include "wombat/strutils.h"
>
> #include <mama/mama.h>
>+#include <mama/msg.h>
> #include <mama/error.h>
> #include <mamainternal.h>
> #include <mama/version.h>
>@@ -168,6 +169,7 @@ typedef struct mamaImpl_
>     mamaPayloadBridge      myPayloads[MAMA_PAYLOAD_MAX];
>     LIB_HANDLE             myBridgeLibraries[MAMA_MIDDLEWARE_MAX];
>     LIB_HANDLE             myPayloadLibraries[MAMA_PAYLOAD_MAX];
>+    msgPayload_destroyImpl myPayloadDestroyFuncs[MAMA_PAYLOAD_MAX];
>     unsigned int           myRefCount;
>     wthread_static_mutex_t myLock;
> } mamaImpl;
>@@ -175,7 +177,7 @@ typedef struct mamaImpl_
> static mamaApplicationContext  appContext;
> static char mama_ver_string[256];
>
>-static mamaImpl gImpl = {{0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>+static mamaImpl gImpl = {{0}, {0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>
> /*
>*************************************************************************
>*/
> /* Private Function Prototypes. */
>@@ -624,10 +626,21 @@ mamaInternal_findBridge ()
> mamaPayloadBridge
> mamaInternal_findPayload (char id)
> {
>+    mamaPayloadBridge bridge = NULL;
>+
>     if ('\0' == id)
>         return NULL;
>
>-    return gImpl.myPayloads[(uint8_t)id];
>+    bridge = gImpl.myPayloads[(uint8_t)id];
>+
>+    if (NULL == bridge)
>+    {
>+        /* Dynamically load the payload bridge just-in-time, if it
>hasn't
>+         * already been loaded previously. */
>+        mama_loadPayloadBridgeByType (&bridge, (mamaPayloadType)id);
>+    }
>+
>+    return bridge;
> }
>
> mamaPayloadBridge
>@@ -1241,17 +1254,33 @@ mama_closeCount (unsigned int* count)
>
>         cleanupReservedFields();
>
>-         /* Look for a bridge for each of the payloads and close them */
>+        /* Look for a bridge for each of the payloads and close them */
>         for (payload = 0; payload != MAMA_PAYLOAD_MAX; ++payload)
>         {
>-              /* mamaPayloadBridgeImpl* impl = (mamaPayloadBridgeImpl*)
>-             * gImpl.myPayloads [(uint8_t)payload];*/
>-            gImpl.myPayloads[(uint8_t)payload] = NULL;
>-            if(gImpl.myPayloadLibraries[(uint8_t)payload])
>+            uint8_t                id   = (uint8_t)payload;
>+            LIB_HANDLE             lib  = gImpl.myPayloadLibraries[id];
>+            mamaPayloadBridgeImpl* impl = gImpl.myPayloads[id];
>+
>+            if (!lib)
>             {
>-                closeSharedLib
>(gImpl.myPayloadLibraries[(uint8_t)payload]);
>-                gImpl.myPayloadLibraries[(uint8_t)payload] = NULL;
>+                continue;
>+            }
>+
>+            if (impl)
>+            {
>+                msgPayload_destroyImpl destroyFunc =
>+                    gImpl.myPayloadDestroyFuncs[id];
>+
>+                if (destroyFunc)
>+                {
>+                    destroyFunc (impl);
>+                }
>+
>+                gImpl.myPayloads[id] = NULL;
>             }
>+
>+            closeSharedLib (lib);
>+            gImpl.myPayloadLibraries[id] = NULL;
>         }
>
>        gDefaultPayload = NULL;
>@@ -1827,18 +1856,29 @@ mama_setDefaultPayload (char id)
>
> mama_status
> mama_loadPayloadBridge (mamaPayloadBridge* impl,
>-                         const char*        payloadName)
>+                        const char*        payloadName)
> {
>     return mama_loadPayloadBridgeInternal (impl, payloadName);
> }
>+
>+mama_status
>+mama_loadPayloadBridgeByType (mamaPayloadBridge* impl,
>+                              mamaPayloadType    payload)
>+{
>+    const char* name = mamaPayload_convertToLibString (payload);
>+    return mama_loadPayloadBridgeInternal (impl, name);
>+}
>+
> mama_status
> mama_loadPayloadBridgeInternal  (mamaPayloadBridge* impl,
>                                  const char*        payloadName)
> {
>     char                    bridgeImplName  [256];
>     char                    initFuncName    [256];
>+    char                    destroyFuncName [256];
>     LIB_HANDLE              bridgeLib       = NULL;
>     msgPayload_createImpl   initFunc        = NULL;
>+    msgPayload_destroyImpl  destroyFunc     = NULL;
>     mama_status             status          = MAMA_STATUS_OK;
>     char                    payloadChar       ='\0';
>       void*                                   vp                              = NULL;
>@@ -1865,11 +1905,10 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>         return MAMA_STATUS_NO_BRIDGE_IMPL;
>     }
>
>-    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>-
>     /* Gives a warning - casting from void* to bridge_createImpl func */
>-      vp = loadLibFunc (bridgeLib, initFuncName);
>-      initFunc  = *(msgPayload_createImpl*) &vp;
>+    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>+    vp = loadLibFunc (bridgeLib, initFuncName);
>+    initFunc  = *(msgPayload_createImpl*) &vp;
>
>     if (!initFunc)
>     {
>@@ -1887,7 +1926,7 @@ mama_loadPayloadBridgeInternal  (mamaPayloadBridge*
>impl,
>
>     if (MAMA_STATUS_OK != (status = initFunc (impl, &payloadChar)))
>     {
>-       wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return status;
>     }
>@@ -1910,13 +1949,19 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>              "Payload bridge %s already loaded",
>              payloadName);
>
>-         wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return MAMA_STATUS_OK;
>     }
>
>+    /* Gives a warning - casting from void* to bridge_destroyImpl func */
>+    snprintf (destroyFuncName, 256, "%sPayload_destroyImpl",
>payloadName);
>+    vp = loadLibFunc (bridgeLib, destroyFuncName);
>+    destroyFunc = *(msgPayload_destroyImpl*) &vp;
>+
>     gImpl.myPayloads [(int)payloadChar] = *impl;
>     gImpl.myPayloadLibraries [(int)payloadChar] = bridgeLib;
>+    gImpl.myPayloadDestroyFuncs [(int)payloadChar] = destroyFunc;
>
>     if (!gDefaultPayload)
>     {
>diff --git a/mama/c_cpp/src/c/mama/mama.h b/mama/c_cpp/src/c/mama/mama.h
>index a574f36..07a9115 100644
>--- a/mama/c_cpp/src/c/mama/mama.h
>+++ b/mama/c_cpp/src/c/mama/mama.h
>@@ -133,11 +133,34 @@ extern "C"
>      */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadBridge (mamaBridge* impl, const char*  middleware);
>+    mama_loadBridge (mamaBridge* impl,
>+                     const char* middleware);
>
>+    /**
>+     * Load a payload bridge by its well-known library name string.
>E.g. "wmsg"
>+     * will load the WombatMsg payload, provided it exists on the system.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payloadName The well-known name of the payload library.
>+     */
>+    MAMAExpDLL
>+    extern mama_status
>+    mama_loadPayloadBridge (mamaPayloadBridge* bridge,
>+                            const char*        payloadName);
>+
>+    /**
>+     * Load a payload bridge by using the mamaPayloadType identifier.
>This
>+     * utilises mamaPayload_convertToLibString() in order to retrieve the
>+     * well-known library string for the payload library, and this is
>then
>+     * delegated to mama_loadPayloadBridge.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payload The payload enumeration identifier.
>+     */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadPayloadBridge (mamaPayloadBridge*  bridge,  const char*
>payloadName);
>+    mama_loadPayloadBridgeByType (mamaPayloadBridge* bridge,
>+                                  mamaPayloadType    payload);
>
>     /** Load the bridge specified by middleware string using the path
>specified by the user.
>      * If the bridge has already been loaded then the existing bridge
>instance
>diff --git a/mama/c_cpp/src/c/mama/msg.h b/mama/c_cpp/src/c/mama/msg.h
>index bf0d562..a9c3438 100644
>--- a/mama/c_cpp/src/c/mama/msg.h
>+++ b/mama/c_cpp/src/c/mama/msg.h
>@@ -74,6 +74,36 @@ extern const char*
> mamaPayload_convertToString (mamaPayloadType payloadType);
>
> /**
>+ * Convert a string to a mamaPayloadType.
>+ *
>+ * @param str The str to convert.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromString (const char* str);
>+
>+/**
>+ * Convert a mamaPayloadType value to a well-known library name string
>+ * for that payload.  E.g. MAMA_PAYLOAD_WOMBAT_MSG translates into
>+ * "wmsg" (of which the payload library would be libwmsgimpl.so on *nix).
>+ *
>+ * @param payloadType The payloadType to convert to a library name
>string.
>+ */
>+MAMAExpDLL
>+extern const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType);
>+
>+/**
>+ * Convert a well-known library name to a mamaPayloadType value
>+ * for that payload.  E.g. "wmsg" translates into
>MAMA_PAYLOAD_WOMBAT_MSG.
>+ *
>+ * @param str The library name string to convert to a mamaPayloadType.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str);
>+
>+/**
>  * Create a mamaMsg.
>  *
>  * @param msg The location of a mamaMsg where to store the result.
>diff --git a/mama/c_cpp/src/c/msg.c b/mama/c_cpp/src/c/msg.c
>index 286fcad..506c390 100644
>--- a/mama/c_cpp/src/c/msg.c
>+++ b/mama/c_cpp/src/c/msg.c
>@@ -41,13 +41,39 @@
>
> #include "wombat/wincompat.h"
>
>-
> #define MAX_SUBJECT 256
>
>-#define NOFID 0
>-
>-static const int INCLUDE_FIELD_NAME = (1 << 4);
>-static const int INCLUDE_FIELD_ID   = (1 << 2);
>+typedef struct PayloadTable
>+{
>+    mamaPayloadType mType;
>+    const char*     mName;
>+    const char*     mLibrary;
>+}
>+PayloadTable;
>+
>+static
>+const PayloadTable kPayloads [] =
>+{
>+    {MAMA_PAYLOAD_ACTIV,      "activ",      "activmsg"},
>+    {MAMA_PAYLOAD_AVIS,       "AVIS",       "avismsg"},
>+    {MAMA_PAYLOAD_EXEGY,      "EXEGY",      "exegymsg"},
>+    {MAMA_PAYLOAD_FAST,       "FAST",       "fastmsg"},
>+    {MAMA_PAYLOAD_IBMWFO,     "ibmwfo",     "ibmwfomsg"},
>+    {MAMA_PAYLOAD_INRUSH,     "INRUSH",     "inrushmsg"},
>+    {MAMA_PAYLOAD_QPID,       "QPID",       "qpidmsg"},
>+    {MAMA_PAYLOAD_RAI,        "rai",        "raimsg"},
>+    {MAMA_PAYLOAD_SOLACE,     "solacemsg",  "solacemsg"},
>+    {MAMA_PAYLOAD_TIBRV,      "TIBRV",      "tibrvmsg"},
>+    {MAMA_PAYLOAD_TICK42BLP,  "TICK42BLP",  "tick42blpmsg"},
>+    {MAMA_PAYLOAD_TICK42RMDS, "TICK42RMDS", "tick42rmdsmsg"},
>+    {MAMA_PAYLOAD_UMS,        "UMS",        "umsmsg"},
>+    {MAMA_PAYLOAD_V5,         "V5",         "wcachemsg"},
>+    {MAMA_PAYLOAD_VULCAN,     "Vulcan",     "vulcanmsg"},
>+    {MAMA_PAYLOAD_WOMBAT_MSG, "WombatMsg",  "wmsg"},
>+
>+    /* Linear search requires unknown to be at the end-of-table */
>+    {MAMA_PAYLOAD_UNKNOWN,    "unknown",    NULL}
>+};
>
> typedef struct mamaMsgIteratorImpl
> {
>@@ -106,6 +132,66 @@ typedef struct mamaMsgImpl_
> static mama_status
> mamaMsgImpl_destroyLastVectorMsg (mamaMsgImpl *impl);
>
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByType (mamaPayloadType payloadType)
>+{
>+    int k = 0;
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (kPayloads[k].mType == payloadType)
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByName (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mName))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByLibrary (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mLibrary))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
> /*=================================================================
>   = Public functions - defined in mama/msg.h
>   ================================================================*/
>@@ -166,7 +252,9 @@ mamaMsg_destroy (mamaMsg msg)
>     if (impl->mCurrentField)
>     {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>     }
>+
>     /*Destroy the reusable field object*/
>     if (impl->mCopy)
>     {
>@@ -366,44 +454,29 @@ mamaMsgImpl_getPayload (const mamaMsg msg,
>msgPayload* payload)
> const char*
> mamaPayload_convertToString (mamaPayloadType payloadType)
> {
>-    switch (payloadType)
>-    {
>-        case MAMA_PAYLOAD_SOLACE:
>-            return "solacemsg";
>-        case MAMA_PAYLOAD_V5:
>-            return "V5";
>-        case MAMA_PAYLOAD_AVIS:
>-            return "AVIS";
>-        case MAMA_PAYLOAD_TICK42BLP:
>-            return "TICK42BLP";
>-        case MAMA_PAYLOAD_FAST:
>-            return "FAST";
>-        case MAMA_PAYLOAD_RAI:
>-            return "rai";
>-        case MAMA_PAYLOAD_UMS:
>-            return "UMS";
>-        case MAMA_PAYLOAD_TICK42RMDS:
>-            return "TICK42RMDS";
>-        case MAMA_PAYLOAD_QPID:
>-            return "QPID";
>-        case MAMA_PAYLOAD_TIBRV:
>-            return "TIBRV";
>-        case MAMA_PAYLOAD_IBMWFO:
>-            return "ibmwfo";
>-        case MAMA_PAYLOAD_ACTIV:
>-            return "activ";
>-        case MAMA_PAYLOAD_VULCAN:
>-            return "Vulcan";
>-        case MAMA_PAYLOAD_WOMBAT_MSG:
>-            return "WombatMsg";
>-        case MAMA_PAYLOAD_EXEGY:
>-            return "EXEGY";
>-        case MAMA_PAYLOAD_INRUSH:
>-            return "INRUSH";
>-
>-        default:
>-            return "unknown";
>-    }
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mName;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByName (str);
>+    return table->mType;
>+}
>+
>+const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mLibrary;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByLibrary
>(str);
>+    return table->mType;
> }
>
> mama_status
>@@ -3481,7 +3554,11 @@ mama_status mamaMsgIterator_destroy
>(mamaMsgIterator iterator)
>     }
>
>     if (impl->mCurrentField)
>+    {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>+    }
>+
>     free (impl);
>     return MAMA_STATUS_OK;
> }
>diff --git a/mama/c_cpp/src/c/payloadbridge.h
>b/mama/c_cpp/src/c/payloadbridge.h
>index 703c1a9..6ad7108 100644
>--- a/mama/c_cpp/src/c/payloadbridge.h
>+++ b/mama/c_cpp/src/c/payloadbridge.h
>@@ -422,8 +422,12 @@ while(0)
>                       \
> typedef mama_status
> (*msgPayload_createImpl)      (mamaPayloadBridge* result, char*
>identifier);
>
>+typedef mama_status
>+(*msgPayload_destroyImpl)     (mamaPayloadBridge result);
>+
> typedef mamaPayloadType
> (*msgPayload_getType)         (void);
>+
> /*===================================================================
>  =              msgPayload bridge function pointers                 =
>  ====================================================================*/
>diff --git a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>index ed18a64..1e14227 100644
>--- a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>+++ b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>@@ -20,14 +20,9 @@ protected:
>
>    virtual void TearDown(void);
>
>-   typedef std::pair <mamaPayloadType, std::string> MamaPayloadPair;
>-   typedef std::map  <mamaPayloadType, std::string> MamaPayloadMapType;
>-
>-   typedef std::pair <mamaMiddleware,  std::string> MamaMiddlewarePair;
>-   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>-
>-   MamaPayloadMapType     payloadTestData;
>-   MamaMiddlewareMapType  middlewareTestData;
>+   typedef std::pair <std::string,     std::string>    PayloadNameLib;
>+   typedef std::map  <mamaPayloadType, PayloadNameLib>
>MamaPayloadMapType;
>+   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>
> private:
>    void CreateTestData();
>@@ -55,100 +50,81 @@ MamaEnumTestsC::TearDown()
> void
> MamaEnumTestsC::CreateTestData()
> {
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_V5,
>"V5"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_AVIS,
>"AVIS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42BLP,
>"TICK42BLP"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_FAST,
>"FAST"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_RAI,
>"rai"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UMS,
>"UMS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42RMDS,
>"TICK42RMDS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_QPID,
>"QPID"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TIBRV,
>"TIBRV"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_IBMWFO,
>"ibmwfo"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_ACTIV,
>"activ"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_VULCAN,
>"Vulcan"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_WOMBAT_MSG,
>"WombatMsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_EXEGY,
>"EXEGY"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_SOLACE,
>"solacemsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UNKNOWN,
>"unknown"));
>-
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_WMW,
>     "wmw"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_LBM,
>     "lbm"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TIBRV,     "tibrv"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_AVIS,
>     "AVIS"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42BLP, "tick42blp"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_SOLACE,    "SOLACE"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_RAI,
>     "rai"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_QPID,
>     "QPID"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_EXEGY,     "exegy"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_IBMWFO,    "ibmwfo"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_ACTIV,     "activ"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42RMDS, "tick42rmds"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_UMS,
>     "ums"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_VULCAN,    "vulcan"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_UNKNOWN,   "unknown"));
> }
>
>-
>-TEST_F (MamaEnumTestsC, testPayloadConvertToString)
>+TEST_F (MamaEnumTestsC, testPayloadStringConversions)
> {
>+    MamaPayloadMapType testData;
>+    testData[MAMA_PAYLOAD_ACTIV]      = PayloadNameLib ("activ",
>"activmsg");
>+    testData[MAMA_PAYLOAD_AVIS]       = PayloadNameLib ("AVIS",
>"avismsg");
>+    testData[MAMA_PAYLOAD_EXEGY]      = PayloadNameLib ("EXEGY",
>"exegymsg");
>+    testData[MAMA_PAYLOAD_FAST]       = PayloadNameLib ("FAST",
>"fastmsg");
>+    testData[MAMA_PAYLOAD_IBMWFO]     = PayloadNameLib ("ibmwfo",
>"ibmwfomsg");
>+    testData[MAMA_PAYLOAD_QPID]       = PayloadNameLib ("QPID",
>"qpidmsg");
>+    testData[MAMA_PAYLOAD_RAI]        = PayloadNameLib ("rai",
>"raimsg");
>+    testData[MAMA_PAYLOAD_SOLACE]     = PayloadNameLib ("solacemsg",
>"solacemsg");
>+    testData[MAMA_PAYLOAD_TIBRV]      = PayloadNameLib ("TIBRV",
>"tibrvmsg");
>+    testData[MAMA_PAYLOAD_TICK42BLP]  = PayloadNameLib ("TICK42BLP",
>"tick42blpmsg");
>+    testData[MAMA_PAYLOAD_TICK42RMDS] = PayloadNameLib ("TICK42RMDS",
>"tick42rmdsmsg");
>+    testData[MAMA_PAYLOAD_UMS]        = PayloadNameLib ("UMS",
>"umsmsg");
>+    testData[MAMA_PAYLOAD_VULCAN]     = PayloadNameLib ("Vulcan",
>"vulcanmsg");
>+    testData[MAMA_PAYLOAD_WOMBAT_MSG] = PayloadNameLib ("WombatMsg",
>"wmsg");
>+    testData[MAMA_PAYLOAD_V5]         = PayloadNameLib ("V5",
>"wcachemsg");
>+    testData[MAMA_PAYLOAD_UNKNOWN]    = PayloadNameLib ("unknown",
>"");
>+
>     MamaPayloadMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = payloadTestData.begin(); itr != payloadTestData.end();
>++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaPayloadType payload   = (*itr).first;
>-       std::string     expected  = (*itr).second;
>-
>-       std::string actual = mamaPayload_convertToString (payload);
>-
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       mamaPayloadType payload = itr->first;
>+       PayloadNameLib& namelib = itr->second;
>+       const char*     name    = namelib.first.c_str();
>+       const char*     library = namelib.second.c_str();
>+
>+       // Override for unknown, mama returns NULL for unknown library
>(so that
>+       // it prevents libraries from trying to load libmamaunknownmsg).
>+       if (MAMA_PAYLOAD_UNKNOWN == payload)
>+       {
>+           library = NULL;
>+       }
>+
>+       // Check name/payload conversion
>+       EXPECT_STREQ(name, mamaPayload_convertToString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromString
>(name));
>+
>+       // Check library/payload conversion
>+       EXPECT_STREQ(library, mamaPayload_convertToLibString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromLibString
>(library));
>     }
>-
>-    ASSERT_EQ( passed, true );
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertToString)
>+TEST_F (MamaEnumTestsC, testMiddlewareStringConversions)
> {
>+    MamaMiddlewareMapType testData;
>+    testData[MAMA_MIDDLEWARE_WMW]        = "wmw";
>+    testData[MAMA_MIDDLEWARE_LBM]        = "lbm";
>+    testData[MAMA_MIDDLEWARE_TIBRV]      = "tibrv";
>+    testData[MAMA_MIDDLEWARE_AVIS]       = "AVIS";
>+    testData[MAMA_MIDDLEWARE_TICK42BLP]  = "tick42blp";
>+    testData[MAMA_MIDDLEWARE_SOLACE]     = "SOLACE";
>+    testData[MAMA_MIDDLEWARE_RAI]        = "rai";
>+    testData[MAMA_MIDDLEWARE_QPID]       = "QPID";
>+    testData[MAMA_MIDDLEWARE_EXEGY]      = "exegy";
>+    testData[MAMA_MIDDLEWARE_IBMWFO]     = "ibmwfo";
>+    testData[MAMA_MIDDLEWARE_ACTIV]      = "activ";
>+    testData[MAMA_MIDDLEWARE_TICK42RMDS] = "tick42rmds";
>+    testData[MAMA_MIDDLEWARE_UMS]        = "ums";
>+    testData[MAMA_MIDDLEWARE_VULCAN]     = "vulcan";
>+    testData[MAMA_MIDDLEWARE_UNKNOWN]    = "unknown";
>     MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); ++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaMiddleware middleware = (*itr).first;
>-       std::string    expected   = (*itr).second;
>-
>-       std::string actual = mamaMiddleware_convertToString (middleware);
>+       mamaMiddleware middleware = itr->first;
>+       const char*    name       = itr->second.c_str();
>
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       EXPECT_STREQ(name, mamaMiddleware_convertToString (middleware));
>+       EXPECT_EQ((char)middleware,
>(char)mamaMiddleware_convertFromString (name));
>     }
>-
>-    ASSERT_EQ (passed, true);
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertFromString)
>-{
>-    MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>-
>-    for (itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); itr++) {
>-
>-        mamaMiddleware expected    = (*itr).first;
>-        std::string    middleware  = (*itr).second;
>-
>-        mamaMiddleware actual      = mamaMiddleware_convertFromString
>(middleware.c_str());
>-
>-        EXPECT_EQ (actual, expected);
>-
>-        if (actual != expected)
>-            passed = false;
>-    }
>-
>-    ASSERT_EQ (passed, true);
>-}
>

________________________________________________________

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.
________________________________________________________




--
Lee Skillen

Vulcan Financial Technologies
51 Malone Road, Belfast, BT9 6RY

Web:     www.vulcanft.com 



--
Lee Skillen

Vulcan Financial Technologies
51 Malone Road, Belfast, BT9 6RY

Office:  +44 (0)28 95 817888
Mobile:  +44 (0)78 41 425152
Web:     www.vulcanft.com 


Damian Maguire <DMaguire@...>
 

Cheers for this Lee, looks good. Definitely good to have you guys starting to contribute as well, welcome to the community ;-)

The test setup sounds reasonable, and is similar to what I had in mind. I know Frank ran a few tests when developing Qpid that involved swapping out payloads, so he may have some tools lying around that we can work with already. I'll take a look and get back to you if we need any further info on it anyway. 

Regarding the acceptance point, the general rule is always going to be that we'll accept code that makes OpenMAMA better, and fits the goals of the project moving forward. This certainly matches the first requirement, and feels like a step in the right direction on the second. The real question is how it sits with the broader roadmap, and what changes are required to pull it in line with that. That sort of discussion is really one for Glenn, so I'll leave it to him to comment further.   

And no worries on the patch front – these things happen from time to time, and it's clearly not through any fault of your own (and expecting you to change git-core to fix the problem feels slightly unreasonable to be honest ;-) ). In this instance the work arounds are fairly straight forward (and numerous) anyway, so it's not a major problem. And just remember, practice makes perfect ;-)

Cheers, 


Damian Maguire – Senior R&D and OpenMAMA Specialist
IntercontinentalExchange | NYSE Technologies
24-26 Adelaide Exchange | Belfast, BT2 8GD
Tel: +44 2890 822 282 (ext: 452161) | Mob: +44 7540 204 077

From: Lee Skillen <lskillen@...>
Date: Thursday, February 20, 2014 5:08 PM
To: Damian Maguire <dmaguire@...>
Cc: "openmama-dev@..." <openmama-dev@...>
Subject: Re: [Openmama-dev] [PATCH] MAMA: Add the capability to load payload libraries just-in-time (JIT).

Hey Damian,

Thanks, we're happy to finally be in a better position to start contributing.   First of all, just so that focus isn't purely on the JIT loading, I wanted to point out that this patch also has fixes for memory leaks, a reworking of the payload conversion unit tests, and a new style for the payload mappings (which we hope will be useful from a maintenance perspective to keep things centralised).

The new unit tests exercise the the string conversion capability and we ran the entire suite to ensure that nothing else has been broken.  Memory leaks are confirmed for presence before patch via valgrind and memory sanitizer and their removal is confirmed after.  We also utilise helgrind and thread sanitizer for threading issues in general, although not for this patch.

The JIT loading mechanism has been tested locally using a modified version of avis and our middleware.   Setup for a simple test would be to load a consuming application (e.g. mamaconsumerc) with any middleware that is payload agnostic, and then send a message utilising a payload that is different than the default payload for that middleware.  Pre-patch this would crash the application, but post-patch it'll load the payload just-in-time.

For testing the receiving bit is fine but none of the testtools applications have the capability to specify a different payload other than the default. So for the sake of demonstration I modified mamaproducerc to be capable of specifying payload on the command-line via -P <payload>.  The test looks like this:

Pre-patch:

$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v
Segmentation fault (core dumped)

Post-patch:
 
$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
[snip]
2014-02-20 16:57:20: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]
 
$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]
Starting to publish

Looking back at the mamaconsumer log we can see that it successfully loaded the vulcanmsg payload JIT :-

2014-02-20 16:58:09: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
Rate: 2 Latency(usec): Low: 13775       Average: 447183 High: 910969    Gaps: 1
Rate: 9 Latency(usec): Low: 34754       Average: 507466 High: 933766    Gaps: 0
Rate: 9 Latency(usec): Low: 56764       Average: 528090 High: 954780    Gaps: 0

Hope that looks good for you guys - Please feel free to experiment on the new branch.  We'd really like to get this code in to the mainline if at all possible, and we've got quite a lot more code to contribute too.

Apologies about the carriage returns in the patch file - I don't personally use Windows and we utilise git-format-patch and git-send-email for the patch formatting and sending.  The patch prior to sending doesn't have any carriage return characters in it, but I looked at one I pulled back and it does have 0x0d 0x0a ("\r\n") for the line endings.  So it seems that either SMTP or git-send-email was the culprit.  I'll have a look into preventing it but in the short-term `git apply --whitespace=fix` should sort it. :-(

Cracking start for submitting patches, eh. :-)

Cheers,
Lee




On 20 February 2014 14:36, Damian Maguire <DMaguire@...> wrote:
Cheers for this Lee, looks very interesting. From a quick look/preliminary
test the changes look solid, but given the size and scope of the change I
think the best thing to do is fire it into it's own branch for a while and
let people check it out for themselves. As such I'm going to stick it into
a feature-jit-payload branch on the main repo. Anyone who's interested can
then have a look and comment on the changes, and we can fix any bugs there
as required.

Regarding testing these changes, have you guys actually got this running
with any of the open source bridges, or do you have any sample testing you
can show us to demonstrate that it both works, and doesn't break the
existing code?

Just a quick note on the patch files you've attached - they seem to have
landed (to me at least) with DOS line endings, which messes with the git
apply step. It's not a big deal, since I can easily enough strip the line
endings, but probably something to watch out for with future submissions
(particularly if you touch any of the unfortunate files that already mix
and match line endings).

I'm also taking a look at the other patches you've sent through, so I'll
get back to you on those.

Cheers,

Damian



On 2/19/14 3:53 PM, "Lee Skillen" <lskillen@...> wrote:

>
>If a library hasn't been previously loaded then OpenMAMA will now attempt
>to load it by converting the payload identifier into a well-known string
>for the payload library.  This behaviour is useful if the payload isn't
>known ahead of time or if lazy loading is desired over enumeration of a
>existing set of libraries.  No changes made to existing API interfaces.
>
>The following functions have been added :-
>
>- mamaPayload_convertToLibString():
>  Maps a mamaPayloadType enumeration to the well-known library name.
>
>- mamaPayload_convertFromLibString():
>  Inverse of mamaPayload_convertToLibString.
>
>- mama_loadPayloadBridgeByType():
>  Takes a mamaPayloadType and utilises mamaPayload_convertToLibString()
>  to retrieve the library name for the payload, and then delegates this
>  to the existing mama_loadPayloadBridge() function.
>
>The string conversions are implemented as a linear search, primarily due
>to these functions not being on the hot path - Their use is prior to a
>library being loaded and (I would imagine) would be seldom used in other
>situations.
>
>Additional changes :-
>
>If msgPayload_destroyImpl has been exposed by the payload library then
>this is now utilised to cleanup the payload upon shutdown properly.  It
>was observed that each payload always leaked memory due to lacking the
>capability to destroy itself.  Individual payloads will still need to
>expose this - Backwards compatible as it won't be called otherwise.
>
>The MamaEnumTestsC set of unit tests have been rewritten to utilise the
>new conversion functionality.  The tests have also been renamed as such:
>
>- testPayloadToString -> testPayloadStringConversions
>- testPayloadFromString -> testPayloadStringConversions
>- testMiddlewareToString -> testMiddlewareStringConversions
>- testMiddlewareFromString -> testMiddlewareStringConversions
>
>Noticed that the ASSERT_EQ/EXPECT_EQ statements in gtest are sometimes
>used incorrectly, in that the expected/actual are reversed (it should be
>expected on the left and actual on the right).
>
>Todo :-
>
>Need to confirm well-known library names with all vendors - Will only be
>important if JIT loading is actually used, but the names should be made
>accurate anyway.
>
>Signed-off-by: Lee Skillen <lskillen@...>
>---
> mama/c_cpp/src/c/mama.c                            |  77 ++++++++--
> mama/c_cpp/src/c/mama/mama.h                       |  27 +++-
> mama/c_cpp/src/c/mama/msg.h                        |  30 ++++
> mama/c_cpp/src/c/msg.c                             | 163
>+++++++++++++++------
> mama/c_cpp/src/c/payloadbridge.h                   |   4 +
> .../src/gunittest/c/payloadmiddlewareidtest.cpp    | 152
>++++++++-----------
> 6 files changed, 304 insertions(+), 149 deletions(-)
>
>diff --git a/mama/c_cpp/src/c/mama.c b/mama/c_cpp/src/c/mama.c
>index 33a0768..3a7d7be 100644
>--- a/mama/c_cpp/src/c/mama.c
>+++ b/mama/c_cpp/src/c/mama.c
>@@ -28,6 +28,7 @@
> #include "wombat/strutils.h"
>
> #include <mama/mama.h>
>+#include <mama/msg.h>
> #include <mama/error.h>
> #include <mamainternal.h>
> #include <mama/version.h>
>@@ -168,6 +169,7 @@ typedef struct mamaImpl_
>     mamaPayloadBridge      myPayloads[MAMA_PAYLOAD_MAX];
>     LIB_HANDLE             myBridgeLibraries[MAMA_MIDDLEWARE_MAX];
>     LIB_HANDLE             myPayloadLibraries[MAMA_PAYLOAD_MAX];
>+    msgPayload_destroyImpl myPayloadDestroyFuncs[MAMA_PAYLOAD_MAX];
>     unsigned int           myRefCount;
>     wthread_static_mutex_t myLock;
> } mamaImpl;
>@@ -175,7 +177,7 @@ typedef struct mamaImpl_
> static mamaApplicationContext  appContext;
> static char mama_ver_string[256];
>
>-static mamaImpl gImpl = {{0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>+static mamaImpl gImpl = {{0}, {0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>
> /*
>*************************************************************************
>*/
> /* Private Function Prototypes. */
>@@ -624,10 +626,21 @@ mamaInternal_findBridge ()
> mamaPayloadBridge
> mamaInternal_findPayload (char id)
> {
>+    mamaPayloadBridge bridge = NULL;
>+
>     if ('\0' == id)
>         return NULL;
>
>-    return gImpl.myPayloads[(uint8_t)id];
>+    bridge = gImpl.myPayloads[(uint8_t)id];
>+
>+    if (NULL == bridge)
>+    {
>+        /* Dynamically load the payload bridge just-in-time, if it
>hasn't
>+         * already been loaded previously. */
>+        mama_loadPayloadBridgeByType (&bridge, (mamaPayloadType)id);
>+    }
>+
>+    return bridge;
> }
>
> mamaPayloadBridge
>@@ -1241,17 +1254,33 @@ mama_closeCount (unsigned int* count)
>
>         cleanupReservedFields();
>
>-         /* Look for a bridge for each of the payloads and close them */
>+        /* Look for a bridge for each of the payloads and close them */
>         for (payload = 0; payload != MAMA_PAYLOAD_MAX; ++payload)
>         {
>-              /* mamaPayloadBridgeImpl* impl = (mamaPayloadBridgeImpl*)
>-             * gImpl.myPayloads [(uint8_t)payload];*/
>-            gImpl.myPayloads[(uint8_t)payload] = NULL;
>-            if(gImpl.myPayloadLibraries[(uint8_t)payload])
>+            uint8_t                id   = (uint8_t)payload;
>+            LIB_HANDLE             lib  = gImpl.myPayloadLibraries[id];
>+            mamaPayloadBridgeImpl* impl = gImpl.myPayloads[id];
>+
>+            if (!lib)
>             {
>-                closeSharedLib
>(gImpl.myPayloadLibraries[(uint8_t)payload]);
>-                gImpl.myPayloadLibraries[(uint8_t)payload] = NULL;
>+                continue;
>+            }
>+
>+            if (impl)
>+            {
>+                msgPayload_destroyImpl destroyFunc =
>+                    gImpl.myPayloadDestroyFuncs[id];
>+
>+                if (destroyFunc)
>+                {
>+                    destroyFunc (impl);
>+                }
>+
>+                gImpl.myPayloads[id] = NULL;
>             }
>+
>+            closeSharedLib (lib);
>+            gImpl.myPayloadLibraries[id] = NULL;
>         }
>
>        gDefaultPayload = NULL;
>@@ -1827,18 +1856,29 @@ mama_setDefaultPayload (char id)
>
> mama_status
> mama_loadPayloadBridge (mamaPayloadBridge* impl,
>-                         const char*        payloadName)
>+                        const char*        payloadName)
> {
>     return mama_loadPayloadBridgeInternal (impl, payloadName);
> }
>+
>+mama_status
>+mama_loadPayloadBridgeByType (mamaPayloadBridge* impl,
>+                              mamaPayloadType    payload)
>+{
>+    const char* name = mamaPayload_convertToLibString (payload);
>+    return mama_loadPayloadBridgeInternal (impl, name);
>+}
>+
> mama_status
> mama_loadPayloadBridgeInternal  (mamaPayloadBridge* impl,
>                                  const char*        payloadName)
> {
>     char                    bridgeImplName  [256];
>     char                    initFuncName    [256];
>+    char                    destroyFuncName [256];
>     LIB_HANDLE              bridgeLib       = NULL;
>     msgPayload_createImpl   initFunc        = NULL;
>+    msgPayload_destroyImpl  destroyFunc     = NULL;
>     mama_status             status          = MAMA_STATUS_OK;
>     char                    payloadChar       ='\0';
>       void*                                   vp                              = NULL;
>@@ -1865,11 +1905,10 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>         return MAMA_STATUS_NO_BRIDGE_IMPL;
>     }
>
>-    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>-
>     /* Gives a warning - casting from void* to bridge_createImpl func */
>-      vp = loadLibFunc (bridgeLib, initFuncName);
>-      initFunc  = *(msgPayload_createImpl*) &vp;
>+    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>+    vp = loadLibFunc (bridgeLib, initFuncName);
>+    initFunc  = *(msgPayload_createImpl*) &vp;
>
>     if (!initFunc)
>     {
>@@ -1887,7 +1926,7 @@ mama_loadPayloadBridgeInternal  (mamaPayloadBridge*
>impl,
>
>     if (MAMA_STATUS_OK != (status = initFunc (impl, &payloadChar)))
>     {
>-       wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return status;
>     }
>@@ -1910,13 +1949,19 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>              "Payload bridge %s already loaded",
>              payloadName);
>
>-         wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return MAMA_STATUS_OK;
>     }
>
>+    /* Gives a warning - casting from void* to bridge_destroyImpl func */
>+    snprintf (destroyFuncName, 256, "%sPayload_destroyImpl",
>payloadName);
>+    vp = loadLibFunc (bridgeLib, destroyFuncName);
>+    destroyFunc = *(msgPayload_destroyImpl*) &vp;
>+
>     gImpl.myPayloads [(int)payloadChar] = *impl;
>     gImpl.myPayloadLibraries [(int)payloadChar] = bridgeLib;
>+    gImpl.myPayloadDestroyFuncs [(int)payloadChar] = destroyFunc;
>
>     if (!gDefaultPayload)
>     {
>diff --git a/mama/c_cpp/src/c/mama/mama.h b/mama/c_cpp/src/c/mama/mama.h
>index a574f36..07a9115 100644
>--- a/mama/c_cpp/src/c/mama/mama.h
>+++ b/mama/c_cpp/src/c/mama/mama.h
>@@ -133,11 +133,34 @@ extern "C"
>      */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadBridge (mamaBridge* impl, const char*  middleware);
>+    mama_loadBridge (mamaBridge* impl,
>+                     const char* middleware);
>
>+    /**
>+     * Load a payload bridge by its well-known library name string.
>E.g. "wmsg"
>+     * will load the WombatMsg payload, provided it exists on the system.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payloadName The well-known name of the payload library.
>+     */
>+    MAMAExpDLL
>+    extern mama_status
>+    mama_loadPayloadBridge (mamaPayloadBridge* bridge,
>+                            const char*        payloadName);
>+
>+    /**
>+     * Load a payload bridge by using the mamaPayloadType identifier.
>This
>+     * utilises mamaPayload_convertToLibString() in order to retrieve the
>+     * well-known library string for the payload library, and this is
>then
>+     * delegated to mama_loadPayloadBridge.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payload The payload enumeration identifier.
>+     */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadPayloadBridge (mamaPayloadBridge*  bridge,  const char*
>payloadName);
>+    mama_loadPayloadBridgeByType (mamaPayloadBridge* bridge,
>+                                  mamaPayloadType    payload);
>
>     /** Load the bridge specified by middleware string using the path
>specified by the user.
>      * If the bridge has already been loaded then the existing bridge
>instance
>diff --git a/mama/c_cpp/src/c/mama/msg.h b/mama/c_cpp/src/c/mama/msg.h
>index bf0d562..a9c3438 100644
>--- a/mama/c_cpp/src/c/mama/msg.h
>+++ b/mama/c_cpp/src/c/mama/msg.h
>@@ -74,6 +74,36 @@ extern const char*
> mamaPayload_convertToString (mamaPayloadType payloadType);
>
> /**
>+ * Convert a string to a mamaPayloadType.
>+ *
>+ * @param str The str to convert.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromString (const char* str);
>+
>+/**
>+ * Convert a mamaPayloadType value to a well-known library name string
>+ * for that payload.  E.g. MAMA_PAYLOAD_WOMBAT_MSG translates into
>+ * "wmsg" (of which the payload library would be libwmsgimpl.so on *nix).
>+ *
>+ * @param payloadType The payloadType to convert to a library name
>string.
>+ */
>+MAMAExpDLL
>+extern const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType);
>+
>+/**
>+ * Convert a well-known library name to a mamaPayloadType value
>+ * for that payload.  E.g. "wmsg" translates into
>MAMA_PAYLOAD_WOMBAT_MSG.
>+ *
>+ * @param str The library name string to convert to a mamaPayloadType.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str);
>+
>+/**
>  * Create a mamaMsg.
>  *
>  * @param msg The location of a mamaMsg where to store the result.
>diff --git a/mama/c_cpp/src/c/msg.c b/mama/c_cpp/src/c/msg.c
>index 286fcad..506c390 100644
>--- a/mama/c_cpp/src/c/msg.c
>+++ b/mama/c_cpp/src/c/msg.c
>@@ -41,13 +41,39 @@
>
> #include "wombat/wincompat.h"
>
>-
> #define MAX_SUBJECT 256
>
>-#define NOFID 0
>-
>-static const int INCLUDE_FIELD_NAME = (1 << 4);
>-static const int INCLUDE_FIELD_ID   = (1 << 2);
>+typedef struct PayloadTable
>+{
>+    mamaPayloadType mType;
>+    const char*     mName;
>+    const char*     mLibrary;
>+}
>+PayloadTable;
>+
>+static
>+const PayloadTable kPayloads [] =
>+{
>+    {MAMA_PAYLOAD_ACTIV,      "activ",      "activmsg"},
>+    {MAMA_PAYLOAD_AVIS,       "AVIS",       "avismsg"},
>+    {MAMA_PAYLOAD_EXEGY,      "EXEGY",      "exegymsg"},
>+    {MAMA_PAYLOAD_FAST,       "FAST",       "fastmsg"},
>+    {MAMA_PAYLOAD_IBMWFO,     "ibmwfo",     "ibmwfomsg"},
>+    {MAMA_PAYLOAD_INRUSH,     "INRUSH",     "inrushmsg"},
>+    {MAMA_PAYLOAD_QPID,       "QPID",       "qpidmsg"},
>+    {MAMA_PAYLOAD_RAI,        "rai",        "raimsg"},
>+    {MAMA_PAYLOAD_SOLACE,     "solacemsg",  "solacemsg"},
>+    {MAMA_PAYLOAD_TIBRV,      "TIBRV",      "tibrvmsg"},
>+    {MAMA_PAYLOAD_TICK42BLP,  "TICK42BLP",  "tick42blpmsg"},
>+    {MAMA_PAYLOAD_TICK42RMDS, "TICK42RMDS", "tick42rmdsmsg"},
>+    {MAMA_PAYLOAD_UMS,        "UMS",        "umsmsg"},
>+    {MAMA_PAYLOAD_V5,         "V5",         "wcachemsg"},
>+    {MAMA_PAYLOAD_VULCAN,     "Vulcan",     "vulcanmsg"},
>+    {MAMA_PAYLOAD_WOMBAT_MSG, "WombatMsg",  "wmsg"},
>+
>+    /* Linear search requires unknown to be at the end-of-table */
>+    {MAMA_PAYLOAD_UNKNOWN,    "unknown",    NULL}
>+};
>
> typedef struct mamaMsgIteratorImpl
> {
>@@ -106,6 +132,66 @@ typedef struct mamaMsgImpl_
> static mama_status
> mamaMsgImpl_destroyLastVectorMsg (mamaMsgImpl *impl);
>
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByType (mamaPayloadType payloadType)
>+{
>+    int k = 0;
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (kPayloads[k].mType == payloadType)
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByName (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mName))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByLibrary (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mLibrary))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
> /*=================================================================
>   = Public functions - defined in mama/msg.h
>   ================================================================*/
>@@ -166,7 +252,9 @@ mamaMsg_destroy (mamaMsg msg)
>     if (impl->mCurrentField)
>     {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>     }
>+
>     /*Destroy the reusable field object*/
>     if (impl->mCopy)
>     {
>@@ -366,44 +454,29 @@ mamaMsgImpl_getPayload (const mamaMsg msg,
>msgPayload* payload)
> const char*
> mamaPayload_convertToString (mamaPayloadType payloadType)
> {
>-    switch (payloadType)
>-    {
>-        case MAMA_PAYLOAD_SOLACE:
>-            return "solacemsg";
>-        case MAMA_PAYLOAD_V5:
>-            return "V5";
>-        case MAMA_PAYLOAD_AVIS:
>-            return "AVIS";
>-        case MAMA_PAYLOAD_TICK42BLP:
>-            return "TICK42BLP";
>-        case MAMA_PAYLOAD_FAST:
>-            return "FAST";
>-        case MAMA_PAYLOAD_RAI:
>-            return "rai";
>-        case MAMA_PAYLOAD_UMS:
>-            return "UMS";
>-        case MAMA_PAYLOAD_TICK42RMDS:
>-            return "TICK42RMDS";
>-        case MAMA_PAYLOAD_QPID:
>-            return "QPID";
>-        case MAMA_PAYLOAD_TIBRV:
>-            return "TIBRV";
>-        case MAMA_PAYLOAD_IBMWFO:
>-            return "ibmwfo";
>-        case MAMA_PAYLOAD_ACTIV:
>-            return "activ";
>-        case MAMA_PAYLOAD_VULCAN:
>-            return "Vulcan";
>-        case MAMA_PAYLOAD_WOMBAT_MSG:
>-            return "WombatMsg";
>-        case MAMA_PAYLOAD_EXEGY:
>-            return "EXEGY";
>-        case MAMA_PAYLOAD_INRUSH:
>-            return "INRUSH";
>-
>-        default:
>-            return "unknown";
>-    }
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mName;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByName (str);
>+    return table->mType;
>+}
>+
>+const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mLibrary;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByLibrary
>(str);
>+    return table->mType;
> }
>
> mama_status
>@@ -3481,7 +3554,11 @@ mama_status mamaMsgIterator_destroy
>(mamaMsgIterator iterator)
>     }
>
>     if (impl->mCurrentField)
>+    {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>+    }
>+
>     free (impl);
>     return MAMA_STATUS_OK;
> }
>diff --git a/mama/c_cpp/src/c/payloadbridge.h
>b/mama/c_cpp/src/c/payloadbridge.h
>index 703c1a9..6ad7108 100644
>--- a/mama/c_cpp/src/c/payloadbridge.h
>+++ b/mama/c_cpp/src/c/payloadbridge.h
>@@ -422,8 +422,12 @@ while(0)
>                       \
> typedef mama_status
> (*msgPayload_createImpl)      (mamaPayloadBridge* result, char*
>identifier);
>
>+typedef mama_status
>+(*msgPayload_destroyImpl)     (mamaPayloadBridge result);
>+
> typedef mamaPayloadType
> (*msgPayload_getType)         (void);
>+
> /*===================================================================
>  =              msgPayload bridge function pointers                 =
>  ====================================================================*/
>diff --git a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>index ed18a64..1e14227 100644
>--- a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>+++ b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>@@ -20,14 +20,9 @@ protected:
>
>    virtual void TearDown(void);
>
>-   typedef std::pair <mamaPayloadType, std::string> MamaPayloadPair;
>-   typedef std::map  <mamaPayloadType, std::string> MamaPayloadMapType;
>-
>-   typedef std::pair <mamaMiddleware,  std::string> MamaMiddlewarePair;
>-   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>-
>-   MamaPayloadMapType     payloadTestData;
>-   MamaMiddlewareMapType  middlewareTestData;
>+   typedef std::pair <std::string,     std::string>    PayloadNameLib;
>+   typedef std::map  <mamaPayloadType, PayloadNameLib>
>MamaPayloadMapType;
>+   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>
> private:
>    void CreateTestData();
>@@ -55,100 +50,81 @@ MamaEnumTestsC::TearDown()
> void
> MamaEnumTestsC::CreateTestData()
> {
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_V5,
>"V5"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_AVIS,
>"AVIS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42BLP,
>"TICK42BLP"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_FAST,
>"FAST"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_RAI,
>"rai"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UMS,
>"UMS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42RMDS,
>"TICK42RMDS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_QPID,
>"QPID"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TIBRV,
>"TIBRV"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_IBMWFO,
>"ibmwfo"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_ACTIV,
>"activ"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_VULCAN,
>"Vulcan"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_WOMBAT_MSG,
>"WombatMsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_EXEGY,
>"EXEGY"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_SOLACE,
>"solacemsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UNKNOWN,
>"unknown"));
>-
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_WMW,
>     "wmw"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_LBM,
>     "lbm"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TIBRV,     "tibrv"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_AVIS,
>     "AVIS"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42BLP, "tick42blp"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_SOLACE,    "SOLACE"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_RAI,
>     "rai"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_QPID,
>     "QPID"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_EXEGY,     "exegy"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_IBMWFO,    "ibmwfo"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_ACTIV,     "activ"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42RMDS, "tick42rmds"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_UMS,
>     "ums"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_VULCAN,    "vulcan"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_UNKNOWN,   "unknown"));
> }
>
>-
>-TEST_F (MamaEnumTestsC, testPayloadConvertToString)
>+TEST_F (MamaEnumTestsC, testPayloadStringConversions)
> {
>+    MamaPayloadMapType testData;
>+    testData[MAMA_PAYLOAD_ACTIV]      = PayloadNameLib ("activ",
>"activmsg");
>+    testData[MAMA_PAYLOAD_AVIS]       = PayloadNameLib ("AVIS",
>"avismsg");
>+    testData[MAMA_PAYLOAD_EXEGY]      = PayloadNameLib ("EXEGY",
>"exegymsg");
>+    testData[MAMA_PAYLOAD_FAST]       = PayloadNameLib ("FAST",
>"fastmsg");
>+    testData[MAMA_PAYLOAD_IBMWFO]     = PayloadNameLib ("ibmwfo",
>"ibmwfomsg");
>+    testData[MAMA_PAYLOAD_QPID]       = PayloadNameLib ("QPID",
>"qpidmsg");
>+    testData[MAMA_PAYLOAD_RAI]        = PayloadNameLib ("rai",
>"raimsg");
>+    testData[MAMA_PAYLOAD_SOLACE]     = PayloadNameLib ("solacemsg",
>"solacemsg");
>+    testData[MAMA_PAYLOAD_TIBRV]      = PayloadNameLib ("TIBRV",
>"tibrvmsg");
>+    testData[MAMA_PAYLOAD_TICK42BLP]  = PayloadNameLib ("TICK42BLP",
>"tick42blpmsg");
>+    testData[MAMA_PAYLOAD_TICK42RMDS] = PayloadNameLib ("TICK42RMDS",
>"tick42rmdsmsg");
>+    testData[MAMA_PAYLOAD_UMS]        = PayloadNameLib ("UMS",
>"umsmsg");
>+    testData[MAMA_PAYLOAD_VULCAN]     = PayloadNameLib ("Vulcan",
>"vulcanmsg");
>+    testData[MAMA_PAYLOAD_WOMBAT_MSG] = PayloadNameLib ("WombatMsg",
>"wmsg");
>+    testData[MAMA_PAYLOAD_V5]         = PayloadNameLib ("V5",
>"wcachemsg");
>+    testData[MAMA_PAYLOAD_UNKNOWN]    = PayloadNameLib ("unknown",
>"");
>+
>     MamaPayloadMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = payloadTestData.begin(); itr != payloadTestData.end();
>++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaPayloadType payload   = (*itr).first;
>-       std::string     expected  = (*itr).second;
>-
>-       std::string actual = mamaPayload_convertToString (payload);
>-
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       mamaPayloadType payload = itr->first;
>+       PayloadNameLib& namelib = itr->second;
>+       const char*     name    = namelib.first.c_str();
>+       const char*     library = namelib.second.c_str();
>+
>+       // Override for unknown, mama returns NULL for unknown library
>(so that
>+       // it prevents libraries from trying to load libmamaunknownmsg).
>+       if (MAMA_PAYLOAD_UNKNOWN == payload)
>+       {
>+           library = NULL;
>+       }
>+
>+       // Check name/payload conversion
>+       EXPECT_STREQ(name, mamaPayload_convertToString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromString
>(name));
>+
>+       // Check library/payload conversion
>+       EXPECT_STREQ(library, mamaPayload_convertToLibString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromLibString
>(library));
>     }
>-
>-    ASSERT_EQ( passed, true );
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertToString)
>+TEST_F (MamaEnumTestsC, testMiddlewareStringConversions)
> {
>+    MamaMiddlewareMapType testData;
>+    testData[MAMA_MIDDLEWARE_WMW]        = "wmw";
>+    testData[MAMA_MIDDLEWARE_LBM]        = "lbm";
>+    testData[MAMA_MIDDLEWARE_TIBRV]      = "tibrv";
>+    testData[MAMA_MIDDLEWARE_AVIS]       = "AVIS";
>+    testData[MAMA_MIDDLEWARE_TICK42BLP]  = "tick42blp";
>+    testData[MAMA_MIDDLEWARE_SOLACE]     = "SOLACE";
>+    testData[MAMA_MIDDLEWARE_RAI]        = "rai";
>+    testData[MAMA_MIDDLEWARE_QPID]       = "QPID";
>+    testData[MAMA_MIDDLEWARE_EXEGY]      = "exegy";
>+    testData[MAMA_MIDDLEWARE_IBMWFO]     = "ibmwfo";
>+    testData[MAMA_MIDDLEWARE_ACTIV]      = "activ";
>+    testData[MAMA_MIDDLEWARE_TICK42RMDS] = "tick42rmds";
>+    testData[MAMA_MIDDLEWARE_UMS]        = "ums";
>+    testData[MAMA_MIDDLEWARE_VULCAN]     = "vulcan";
>+    testData[MAMA_MIDDLEWARE_UNKNOWN]    = "unknown";
>     MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); ++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaMiddleware middleware = (*itr).first;
>-       std::string    expected   = (*itr).second;
>-
>-       std::string actual = mamaMiddleware_convertToString (middleware);
>+       mamaMiddleware middleware = itr->first;
>+       const char*    name       = itr->second.c_str();
>
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       EXPECT_STREQ(name, mamaMiddleware_convertToString (middleware));
>+       EXPECT_EQ((char)middleware,
>(char)mamaMiddleware_convertFromString (name));
>     }
>-
>-    ASSERT_EQ (passed, true);
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertFromString)
>-{
>-    MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>-
>-    for (itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); itr++) {
>-
>-        mamaMiddleware expected    = (*itr).first;
>-        std::string    middleware  = (*itr).second;
>-
>-        mamaMiddleware actual      = mamaMiddleware_convertFromString
>(middleware.c_str());
>-
>-        EXPECT_EQ (actual, expected);
>-
>-        if (actual != expected)
>-            passed = false;
>-    }
>-
>-    ASSERT_EQ (passed, true);
>-}
>

________________________________________________________

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.
________________________________________________________




--
Lee Skillen

Vulcan Financial Technologies
51 Malone Road, Belfast, BT9 6RY

Office:  +44 (0)28 95 817888
Mobile:  +44 (0)78 41 425152
Web:     www.vulcanft.com 

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


Glenn McClements <gmcclements@...>
 

Hi Lee,
At first glance the code looks good, and this is nice little feature with no adverse effects that I can think of. My only issue is that if we're tinkering with this area we should:
  • Go the whole hog and make payloads completely dynamic, removing the enum dependancy. Something like point OpenMAMA to a directory and it automatically loading all payloads in there, and query the payload on startup for it's identifier, not allowing two with the same identifier. There's some security issues with this I suppose.
  • Make it applicable to middleware (and any future kind of) bridges as well.

Happy to hear other peoples opinion on this. 

Cheers, 
Glenn 

From: Lee Skillen <lskillen@...>
Date: Thursday, 20 February 2014 13:42
To: Damian Maguire <DMaguire@...>
Cc: "openmama-dev@..." <openmama-dev@...>
Subject: Re: [Openmama-dev] [PATCH] MAMA: Add the capability to load payload libraries just-in-time (JIT).

Hey,

Just a quick update, it looks like our SMTP server caused the carriage returns - I've modified a version of git-send-email to include the capability to mime encode attachments as base64.  Tested and it carries through the patch in verbatim, so I shouldn't be sending any more patches that look like they came from a Windows machine. ;-)  I'll tidy up the git patch and submit it back to git-core next week since it seems useful (and may be useful to others here too).  Let me know if you need to me to re-submit the OpenMAMA patches I've already sent.

Thanks,
Lee



On 20 February 2014 17:08, Lee Skillen <lskillen@...> wrote:
Hey Damian,

Thanks, we're happy to finally be in a better position to start contributing.   First of all, just so that focus isn't purely on the JIT loading, I wanted to point out that this patch also has fixes for memory leaks, a reworking of the payload conversion unit tests, and a new style for the payload mappings (which we hope will be useful from a maintenance perspective to keep things centralised).

The new unit tests exercise the the string conversion capability and we ran the entire suite to ensure that nothing else has been broken.  Memory leaks are confirmed for presence before patch via valgrind and memory sanitizer and their removal is confirmed after.  We also utilise helgrind and thread sanitizer for threading issues in general, although not for this patch.

The JIT loading mechanism has been tested locally using a modified version of avis and our middleware.   Setup for a simple test would be to load a consuming application (e.g. mamaconsumerc) with any middleware that is payload agnostic, and then send a message utilising a payload that is different than the default payload for that middleware.  Pre-patch this would crash the application, but post-patch it'll load the payload just-in-time.

For testing the receiving bit is fine but none of the testtools applications have the capability to specify a different payload other than the default. So for the sake of demonstration I modified mamaproducerc to be capable of specifying payload on the command-line via -P <payload>.  The test looks like this:

Pre-patch:

$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v
Segmentation fault (core dumped)

Post-patch:
 
$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
[snip]
2014-02-20 16:57:20: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]
 
$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]
Starting to publish

Looking back at the mamaconsumer log we can see that it successfully loaded the vulcanmsg payload JIT :-

2014-02-20 16:58:09: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
Rate: 2 Latency(usec): Low: 13775       Average: 447183 High: 910969    Gaps: 1
Rate: 9 Latency(usec): Low: 34754       Average: 507466 High: 933766    Gaps: 0
Rate: 9 Latency(usec): Low: 56764       Average: 528090 High: 954780    Gaps: 0

Hope that looks good for you guys - Please feel free to experiment on the new branch.  We'd really like to get this code in to the mainline if at all possible, and we've got quite a lot more code to contribute too.

Apologies about the carriage returns in the patch file - I don't personally use Windows and we utilise git-format-patch and git-send-email for the patch formatting and sending.  The patch prior to sending doesn't have any carriage return characters in it, but I looked at one I pulled back and it does have 0x0d 0x0a ("\r\n") for the line endings.  So it seems that either SMTP or git-send-email was the culprit.  I'll have a look into preventing it but in the short-term `git apply --whitespace=fix` should sort it. :-(

Cracking start for submitting patches, eh. :-)

Cheers,
Lee




On 20 February 2014 14:36, Damian Maguire <DMaguire@...> wrote:
Cheers for this Lee, looks very interesting. From a quick look/preliminary
test the changes look solid, but given the size and scope of the change I
think the best thing to do is fire it into it's own branch for a while and
let people check it out for themselves. As such I'm going to stick it into
a feature-jit-payload branch on the main repo. Anyone who's interested can
then have a look and comment on the changes, and we can fix any bugs there
as required.

Regarding testing these changes, have you guys actually got this running
with any of the open source bridges, or do you have any sample testing you
can show us to demonstrate that it both works, and doesn't break the
existing code?

Just a quick note on the patch files you've attached - they seem to have
landed (to me at least) with DOS line endings, which messes with the git
apply step. It's not a big deal, since I can easily enough strip the line
endings, but probably something to watch out for with future submissions
(particularly if you touch any of the unfortunate files that already mix
and match line endings).

I'm also taking a look at the other patches you've sent through, so I'll
get back to you on those.

Cheers,

Damian



On 2/19/14 3:53 PM, "Lee Skillen" <lskillen@...> wrote:

>
>If a library hasn't been previously loaded then OpenMAMA will now attempt
>to load it by converting the payload identifier into a well-known string
>for the payload library.  This behaviour is useful if the payload isn't
>known ahead of time or if lazy loading is desired over enumeration of a
>existing set of libraries.  No changes made to existing API interfaces.
>
>The following functions have been added :-
>
>- mamaPayload_convertToLibString():
>  Maps a mamaPayloadType enumeration to the well-known library name.
>
>- mamaPayload_convertFromLibString():
>  Inverse of mamaPayload_convertToLibString.
>
>- mama_loadPayloadBridgeByType():
>  Takes a mamaPayloadType and utilises mamaPayload_convertToLibString()
>  to retrieve the library name for the payload, and then delegates this
>  to the existing mama_loadPayloadBridge() function.
>
>The string conversions are implemented as a linear search, primarily due
>to these functions not being on the hot path - Their use is prior to a
>library being loaded and (I would imagine) would be seldom used in other
>situations.
>
>Additional changes :-
>
>If msgPayload_destroyImpl has been exposed by the payload library then
>this is now utilised to cleanup the payload upon shutdown properly.  It
>was observed that each payload always leaked memory due to lacking the
>capability to destroy itself.  Individual payloads will still need to
>expose this - Backwards compatible as it won't be called otherwise.
>
>The MamaEnumTestsC set of unit tests have been rewritten to utilise the
>new conversion functionality.  The tests have also been renamed as such:
>
>- testPayloadToString -> testPayloadStringConversions
>- testPayloadFromString -> testPayloadStringConversions
>- testMiddlewareToString -> testMiddlewareStringConversions
>- testMiddlewareFromString -> testMiddlewareStringConversions
>
>Noticed that the ASSERT_EQ/EXPECT_EQ statements in gtest are sometimes
>used incorrectly, in that the expected/actual are reversed (it should be
>expected on the left and actual on the right).
>
>Todo :-
>
>Need to confirm well-known library names with all vendors - Will only be
>important if JIT loading is actually used, but the names should be made
>accurate anyway.
>
>Signed-off-by: Lee Skillen <lskillen@...>
>---
> mama/c_cpp/src/c/mama.c                            |  77 ++++++++--
> mama/c_cpp/src/c/mama/mama.h                       |  27 +++-
> mama/c_cpp/src/c/mama/msg.h                        |  30 ++++
> mama/c_cpp/src/c/msg.c                             | 163
>+++++++++++++++------
> mama/c_cpp/src/c/payloadbridge.h                   |   4 +
> .../src/gunittest/c/payloadmiddlewareidtest.cpp    | 152
>++++++++-----------
> 6 files changed, 304 insertions(+), 149 deletions(-)
>
>diff --git a/mama/c_cpp/src/c/mama.c b/mama/c_cpp/src/c/mama.c
>index 33a0768..3a7d7be 100644
>--- a/mama/c_cpp/src/c/mama.c
>+++ b/mama/c_cpp/src/c/mama.c
>@@ -28,6 +28,7 @@
> #include "wombat/strutils.h"
>
> #include <mama/mama.h>
>+#include <mama/msg.h>
> #include <mama/error.h>
> #include <mamainternal.h>
> #include <mama/version.h>
>@@ -168,6 +169,7 @@ typedef struct mamaImpl_
>     mamaPayloadBridge      myPayloads[MAMA_PAYLOAD_MAX];
>     LIB_HANDLE             myBridgeLibraries[MAMA_MIDDLEWARE_MAX];
>     LIB_HANDLE             myPayloadLibraries[MAMA_PAYLOAD_MAX];
>+    msgPayload_destroyImpl myPayloadDestroyFuncs[MAMA_PAYLOAD_MAX];
>     unsigned int           myRefCount;
>     wthread_static_mutex_t myLock;
> } mamaImpl;
>@@ -175,7 +177,7 @@ typedef struct mamaImpl_
> static mamaApplicationContext  appContext;
> static char mama_ver_string[256];
>
>-static mamaImpl gImpl = {{0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>+static mamaImpl gImpl = {{0}, {0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>
> /*
>*************************************************************************
>*/
> /* Private Function Prototypes. */
>@@ -624,10 +626,21 @@ mamaInternal_findBridge ()
> mamaPayloadBridge
> mamaInternal_findPayload (char id)
> {
>+    mamaPayloadBridge bridge = NULL;
>+
>     if ('\0' == id)
>         return NULL;
>
>-    return gImpl.myPayloads[(uint8_t)id];
>+    bridge = gImpl.myPayloads[(uint8_t)id];
>+
>+    if (NULL == bridge)
>+    {
>+        /* Dynamically load the payload bridge just-in-time, if it
>hasn't
>+         * already been loaded previously. */
>+        mama_loadPayloadBridgeByType (&bridge, (mamaPayloadType)id);
>+    }
>+
>+    return bridge;
> }
>
> mamaPayloadBridge
>@@ -1241,17 +1254,33 @@ mama_closeCount (unsigned int* count)
>
>         cleanupReservedFields();
>
>-         /* Look for a bridge for each of the payloads and close them */
>+        /* Look for a bridge for each of the payloads and close them */
>         for (payload = 0; payload != MAMA_PAYLOAD_MAX; ++payload)
>         {
>-              /* mamaPayloadBridgeImpl* impl = (mamaPayloadBridgeImpl*)
>-             * gImpl.myPayloads [(uint8_t)payload];*/
>-            gImpl.myPayloads[(uint8_t)payload] = NULL;
>-            if(gImpl.myPayloadLibraries[(uint8_t)payload])
>+            uint8_t                id   = (uint8_t)payload;
>+            LIB_HANDLE             lib  = gImpl.myPayloadLibraries[id];
>+            mamaPayloadBridgeImpl* impl = gImpl.myPayloads[id];
>+
>+            if (!lib)
>             {
>-                closeSharedLib
>(gImpl.myPayloadLibraries[(uint8_t)payload]);
>-                gImpl.myPayloadLibraries[(uint8_t)payload] = NULL;
>+                continue;
>+            }
>+
>+            if (impl)
>+            {
>+                msgPayload_destroyImpl destroyFunc =
>+                    gImpl.myPayloadDestroyFuncs[id];
>+
>+                if (destroyFunc)
>+                {
>+                    destroyFunc (impl);
>+                }
>+
>+                gImpl.myPayloads[id] = NULL;
>             }
>+
>+            closeSharedLib (lib);
>+            gImpl.myPayloadLibraries[id] = NULL;
>         }
>
>        gDefaultPayload = NULL;
>@@ -1827,18 +1856,29 @@ mama_setDefaultPayload (char id)
>
> mama_status
> mama_loadPayloadBridge (mamaPayloadBridge* impl,
>-                         const char*        payloadName)
>+                        const char*        payloadName)
> {
>     return mama_loadPayloadBridgeInternal (impl, payloadName);
> }
>+
>+mama_status
>+mama_loadPayloadBridgeByType (mamaPayloadBridge* impl,
>+                              mamaPayloadType    payload)
>+{
>+    const char* name = mamaPayload_convertToLibString (payload);
>+    return mama_loadPayloadBridgeInternal (impl, name);
>+}
>+
> mama_status
> mama_loadPayloadBridgeInternal  (mamaPayloadBridge* impl,
>                                  const char*        payloadName)
> {
>     char                    bridgeImplName  [256];
>     char                    initFuncName    [256];
>+    char                    destroyFuncName [256];
>     LIB_HANDLE              bridgeLib       = NULL;
>     msgPayload_createImpl   initFunc        = NULL;
>+    msgPayload_destroyImpl  destroyFunc     = NULL;
>     mama_status             status          = MAMA_STATUS_OK;
>     char                    payloadChar       ='\0';
>       void*                                   vp                              = NULL;
>@@ -1865,11 +1905,10 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>         return MAMA_STATUS_NO_BRIDGE_IMPL;
>     }
>
>-    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>-
>     /* Gives a warning - casting from void* to bridge_createImpl func */
>-      vp = loadLibFunc (bridgeLib, initFuncName);
>-      initFunc  = *(msgPayload_createImpl*) &vp;
>+    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>+    vp = loadLibFunc (bridgeLib, initFuncName);
>+    initFunc  = *(msgPayload_createImpl*) &vp;
>
>     if (!initFunc)
>     {
>@@ -1887,7 +1926,7 @@ mama_loadPayloadBridgeInternal  (mamaPayloadBridge*
>impl,
>
>     if (MAMA_STATUS_OK != (status = initFunc (impl, &payloadChar)))
>     {
>-       wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return status;
>     }
>@@ -1910,13 +1949,19 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>              "Payload bridge %s already loaded",
>              payloadName);
>
>-         wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return MAMA_STATUS_OK;
>     }
>
>+    /* Gives a warning - casting from void* to bridge_destroyImpl func */
>+    snprintf (destroyFuncName, 256, "%sPayload_destroyImpl",
>payloadName);
>+    vp = loadLibFunc (bridgeLib, destroyFuncName);
>+    destroyFunc = *(msgPayload_destroyImpl*) &vp;
>+
>     gImpl.myPayloads [(int)payloadChar] = *impl;
>     gImpl.myPayloadLibraries [(int)payloadChar] = bridgeLib;
>+    gImpl.myPayloadDestroyFuncs [(int)payloadChar] = destroyFunc;
>
>     if (!gDefaultPayload)
>     {
>diff --git a/mama/c_cpp/src/c/mama/mama.h b/mama/c_cpp/src/c/mama/mama.h
>index a574f36..07a9115 100644
>--- a/mama/c_cpp/src/c/mama/mama.h
>+++ b/mama/c_cpp/src/c/mama/mama.h
>@@ -133,11 +133,34 @@ extern "C"
>      */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadBridge (mamaBridge* impl, const char*  middleware);
>+    mama_loadBridge (mamaBridge* impl,
>+                     const char* middleware);
>
>+    /**
>+     * Load a payload bridge by its well-known library name string.
>E.g. "wmsg"
>+     * will load the WombatMsg payload, provided it exists on the system.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payloadName The well-known name of the payload library.
>+     */
>+    MAMAExpDLL
>+    extern mama_status
>+    mama_loadPayloadBridge (mamaPayloadBridge* bridge,
>+                            const char*        payloadName);
>+
>+    /**
>+     * Load a payload bridge by using the mamaPayloadType identifier.
>This
>+     * utilises mamaPayload_convertToLibString() in order to retrieve the
>+     * well-known library string for the payload library, and this is
>then
>+     * delegated to mama_loadPayloadBridge.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payload The payload enumeration identifier.
>+     */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadPayloadBridge (mamaPayloadBridge*  bridge,  const char*
>payloadName);
>+    mama_loadPayloadBridgeByType (mamaPayloadBridge* bridge,
>+                                  mamaPayloadType    payload);
>
>     /** Load the bridge specified by middleware string using the path
>specified by the user.
>      * If the bridge has already been loaded then the existing bridge
>instance
>diff --git a/mama/c_cpp/src/c/mama/msg.h b/mama/c_cpp/src/c/mama/msg.h
>index bf0d562..a9c3438 100644
>--- a/mama/c_cpp/src/c/mama/msg.h
>+++ b/mama/c_cpp/src/c/mama/msg.h
>@@ -74,6 +74,36 @@ extern const char*
> mamaPayload_convertToString (mamaPayloadType payloadType);
>
> /**
>+ * Convert a string to a mamaPayloadType.
>+ *
>+ * @param str The str to convert.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromString (const char* str);
>+
>+/**
>+ * Convert a mamaPayloadType value to a well-known library name string
>+ * for that payload.  E.g. MAMA_PAYLOAD_WOMBAT_MSG translates into
>+ * "wmsg" (of which the payload library would be libwmsgimpl.so on *nix).
>+ *
>+ * @param payloadType The payloadType to convert to a library name
>string.
>+ */
>+MAMAExpDLL
>+extern const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType);
>+
>+/**
>+ * Convert a well-known library name to a mamaPayloadType value
>+ * for that payload.  E.g. "wmsg" translates into
>MAMA_PAYLOAD_WOMBAT_MSG.
>+ *
>+ * @param str The library name string to convert to a mamaPayloadType.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str);
>+
>+/**
>  * Create a mamaMsg.
>  *
>  * @param msg The location of a mamaMsg where to store the result.
>diff --git a/mama/c_cpp/src/c/msg.c b/mama/c_cpp/src/c/msg.c
>index 286fcad..506c390 100644
>--- a/mama/c_cpp/src/c/msg.c
>+++ b/mama/c_cpp/src/c/msg.c
>@@ -41,13 +41,39 @@
>
> #include "wombat/wincompat.h"
>
>-
> #define MAX_SUBJECT 256
>
>-#define NOFID 0
>-
>-static const int INCLUDE_FIELD_NAME = (1 << 4);
>-static const int INCLUDE_FIELD_ID   = (1 << 2);
>+typedef struct PayloadTable
>+{
>+    mamaPayloadType mType;
>+    const char*     mName;
>+    const char*     mLibrary;
>+}
>+PayloadTable;
>+
>+static
>+const PayloadTable kPayloads [] =
>+{
>+    {MAMA_PAYLOAD_ACTIV,      "activ",      "activmsg"},
>+    {MAMA_PAYLOAD_AVIS,       "AVIS",       "avismsg"},
>+    {MAMA_PAYLOAD_EXEGY,      "EXEGY",      "exegymsg"},
>+    {MAMA_PAYLOAD_FAST,       "FAST",       "fastmsg"},
>+    {MAMA_PAYLOAD_IBMWFO,     "ibmwfo",     "ibmwfomsg"},
>+    {MAMA_PAYLOAD_INRUSH,     "INRUSH",     "inrushmsg"},
>+    {MAMA_PAYLOAD_QPID,       "QPID",       "qpidmsg"},
>+    {MAMA_PAYLOAD_RAI,        "rai",        "raimsg"},
>+    {MAMA_PAYLOAD_SOLACE,     "solacemsg",  "solacemsg"},
>+    {MAMA_PAYLOAD_TIBRV,      "TIBRV",      "tibrvmsg"},
>+    {MAMA_PAYLOAD_TICK42BLP,  "TICK42BLP",  "tick42blpmsg"},
>+    {MAMA_PAYLOAD_TICK42RMDS, "TICK42RMDS", "tick42rmdsmsg"},
>+    {MAMA_PAYLOAD_UMS,        "UMS",        "umsmsg"},
>+    {MAMA_PAYLOAD_V5,         "V5",         "wcachemsg"},
>+    {MAMA_PAYLOAD_VULCAN,     "Vulcan",     "vulcanmsg"},
>+    {MAMA_PAYLOAD_WOMBAT_MSG, "WombatMsg",  "wmsg"},
>+
>+    /* Linear search requires unknown to be at the end-of-table */
>+    {MAMA_PAYLOAD_UNKNOWN,    "unknown",    NULL}
>+};
>
> typedef struct mamaMsgIteratorImpl
> {
>@@ -106,6 +132,66 @@ typedef struct mamaMsgImpl_
> static mama_status
> mamaMsgImpl_destroyLastVectorMsg (mamaMsgImpl *impl);
>
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByType (mamaPayloadType payloadType)
>+{
>+    int k = 0;
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (kPayloads[k].mType == payloadType)
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByName (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mName))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByLibrary (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mLibrary))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
> /*=================================================================
>   = Public functions - defined in mama/msg.h
>   ================================================================*/
>@@ -166,7 +252,9 @@ mamaMsg_destroy (mamaMsg msg)
>     if (impl->mCurrentField)
>     {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>     }
>+
>     /*Destroy the reusable field object*/
>     if (impl->mCopy)
>     {
>@@ -366,44 +454,29 @@ mamaMsgImpl_getPayload (const mamaMsg msg,
>msgPayload* payload)
> const char*
> mamaPayload_convertToString (mamaPayloadType payloadType)
> {
>-    switch (payloadType)
>-    {
>-        case MAMA_PAYLOAD_SOLACE:
>-            return "solacemsg";
>-        case MAMA_PAYLOAD_V5:
>-            return "V5";
>-        case MAMA_PAYLOAD_AVIS:
>-            return "AVIS";
>-        case MAMA_PAYLOAD_TICK42BLP:
>-            return "TICK42BLP";
>-        case MAMA_PAYLOAD_FAST:
>-            return "FAST";
>-        case MAMA_PAYLOAD_RAI:
>-            return "rai";
>-        case MAMA_PAYLOAD_UMS:
>-            return "UMS";
>-        case MAMA_PAYLOAD_TICK42RMDS:
>-            return "TICK42RMDS";
>-        case MAMA_PAYLOAD_QPID:
>-            return "QPID";
>-        case MAMA_PAYLOAD_TIBRV:
>-            return "TIBRV";
>-        case MAMA_PAYLOAD_IBMWFO:
>-            return "ibmwfo";
>-        case MAMA_PAYLOAD_ACTIV:
>-            return "activ";
>-        case MAMA_PAYLOAD_VULCAN:
>-            return "Vulcan";
>-        case MAMA_PAYLOAD_WOMBAT_MSG:
>-            return "WombatMsg";
>-        case MAMA_PAYLOAD_EXEGY:
>-            return "EXEGY";
>-        case MAMA_PAYLOAD_INRUSH:
>-            return "INRUSH";
>-
>-        default:
>-            return "unknown";
>-    }
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mName;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByName (str);
>+    return table->mType;
>+}
>+
>+const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mLibrary;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByLibrary
>(str);
>+    return table->mType;
> }
>
> mama_status
>@@ -3481,7 +3554,11 @@ mama_status mamaMsgIterator_destroy
>(mamaMsgIterator iterator)
>     }
>
>     if (impl->mCurrentField)
>+    {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>+    }
>+
>     free (impl);
>     return MAMA_STATUS_OK;
> }
>diff --git a/mama/c_cpp/src/c/payloadbridge.h
>b/mama/c_cpp/src/c/payloadbridge.h
>index 703c1a9..6ad7108 100644
>--- a/mama/c_cpp/src/c/payloadbridge.h
>+++ b/mama/c_cpp/src/c/payloadbridge.h
>@@ -422,8 +422,12 @@ while(0)
>                       \
> typedef mama_status
> (*msgPayload_createImpl)      (mamaPayloadBridge* result, char*
>identifier);
>
>+typedef mama_status
>+(*msgPayload_destroyImpl)     (mamaPayloadBridge result);
>+
> typedef mamaPayloadType
> (*msgPayload_getType)         (void);
>+
> /*===================================================================
>  =              msgPayload bridge function pointers                 =
>  ====================================================================*/
>diff --git a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>index ed18a64..1e14227 100644
>--- a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>+++ b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>@@ -20,14 +20,9 @@ protected:
>
>    virtual void TearDown(void);
>
>-   typedef std::pair <mamaPayloadType, std::string> MamaPayloadPair;
>-   typedef std::map  <mamaPayloadType, std::string> MamaPayloadMapType;
>-
>-   typedef std::pair <mamaMiddleware,  std::string> MamaMiddlewarePair;
>-   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>-
>-   MamaPayloadMapType     payloadTestData;
>-   MamaMiddlewareMapType  middlewareTestData;
>+   typedef std::pair <std::string,     std::string>    PayloadNameLib;
>+   typedef std::map  <mamaPayloadType, PayloadNameLib>
>MamaPayloadMapType;
>+   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>
> private:
>    void CreateTestData();
>@@ -55,100 +50,81 @@ MamaEnumTestsC::TearDown()
> void
> MamaEnumTestsC::CreateTestData()
> {
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_V5,
>"V5"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_AVIS,
>"AVIS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42BLP,
>"TICK42BLP"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_FAST,
>"FAST"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_RAI,
>"rai"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UMS,
>"UMS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42RMDS,
>"TICK42RMDS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_QPID,
>"QPID"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TIBRV,
>"TIBRV"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_IBMWFO,
>"ibmwfo"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_ACTIV,
>"activ"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_VULCAN,
>"Vulcan"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_WOMBAT_MSG,
>"WombatMsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_EXEGY,
>"EXEGY"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_SOLACE,
>"solacemsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UNKNOWN,
>"unknown"));
>-
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_WMW,
>     "wmw"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_LBM,
>     "lbm"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TIBRV,     "tibrv"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_AVIS,
>     "AVIS"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42BLP, "tick42blp"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_SOLACE,    "SOLACE"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_RAI,
>     "rai"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_QPID,
>     "QPID"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_EXEGY,     "exegy"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_IBMWFO,    "ibmwfo"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_ACTIV,     "activ"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42RMDS, "tick42rmds"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_UMS,
>     "ums"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_VULCAN,    "vulcan"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_UNKNOWN,   "unknown"));
> }
>
>-
>-TEST_F (MamaEnumTestsC, testPayloadConvertToString)
>+TEST_F (MamaEnumTestsC, testPayloadStringConversions)
> {
>+    MamaPayloadMapType testData;
>+    testData[MAMA_PAYLOAD_ACTIV]      = PayloadNameLib ("activ",
>"activmsg");
>+    testData[MAMA_PAYLOAD_AVIS]       = PayloadNameLib ("AVIS",
>"avismsg");
>+    testData[MAMA_PAYLOAD_EXEGY]      = PayloadNameLib ("EXEGY",
>"exegymsg");
>+    testData[MAMA_PAYLOAD_FAST]       = PayloadNameLib ("FAST",
>"fastmsg");
>+    testData[MAMA_PAYLOAD_IBMWFO]     = PayloadNameLib ("ibmwfo",
>"ibmwfomsg");
>+    testData[MAMA_PAYLOAD_QPID]       = PayloadNameLib ("QPID",
>"qpidmsg");
>+    testData[MAMA_PAYLOAD_RAI]        = PayloadNameLib ("rai",
>"raimsg");
>+    testData[MAMA_PAYLOAD_SOLACE]     = PayloadNameLib ("solacemsg",
>"solacemsg");
>+    testData[MAMA_PAYLOAD_TIBRV]      = PayloadNameLib ("TIBRV",
>"tibrvmsg");
>+    testData[MAMA_PAYLOAD_TICK42BLP]  = PayloadNameLib ("TICK42BLP",
>"tick42blpmsg");
>+    testData[MAMA_PAYLOAD_TICK42RMDS] = PayloadNameLib ("TICK42RMDS",
>"tick42rmdsmsg");
>+    testData[MAMA_PAYLOAD_UMS]        = PayloadNameLib ("UMS",
>"umsmsg");
>+    testData[MAMA_PAYLOAD_VULCAN]     = PayloadNameLib ("Vulcan",
>"vulcanmsg");
>+    testData[MAMA_PAYLOAD_WOMBAT_MSG] = PayloadNameLib ("WombatMsg",
>"wmsg");
>+    testData[MAMA_PAYLOAD_V5]         = PayloadNameLib ("V5",
>"wcachemsg");
>+    testData[MAMA_PAYLOAD_UNKNOWN]    = PayloadNameLib ("unknown",
>"");
>+
>     MamaPayloadMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = payloadTestData.begin(); itr != payloadTestData.end();
>++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaPayloadType payload   = (*itr).first;
>-       std::string     expected  = (*itr).second;
>-
>-       std::string actual = mamaPayload_convertToString (payload);
>-
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       mamaPayloadType payload = itr->first;
>+       PayloadNameLib& namelib = itr->second;
>+       const char*     name    = namelib.first.c_str();
>+       const char*     library = namelib.second.c_str();
>+
>+       // Override for unknown, mama returns NULL for unknown library
>(so that
>+       // it prevents libraries from trying to load libmamaunknownmsg).
>+       if (MAMA_PAYLOAD_UNKNOWN == payload)
>+       {
>+           library = NULL;
>+       }
>+
>+       // Check name/payload conversion
>+       EXPECT_STREQ(name, mamaPayload_convertToString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromString
>(name));
>+
>+       // Check library/payload conversion
>+       EXPECT_STREQ(library, mamaPayload_convertToLibString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromLibString
>(library));
>     }
>-
>-    ASSERT_EQ( passed, true );
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertToString)
>+TEST_F (MamaEnumTestsC, testMiddlewareStringConversions)
> {
>+    MamaMiddlewareMapType testData;
>+    testData[MAMA_MIDDLEWARE_WMW]        = "wmw";
>+    testData[MAMA_MIDDLEWARE_LBM]        = "lbm";
>+    testData[MAMA_MIDDLEWARE_TIBRV]      = "tibrv";
>+    testData[MAMA_MIDDLEWARE_AVIS]       = "AVIS";
>+    testData[MAMA_MIDDLEWARE_TICK42BLP]  = "tick42blp";
>+    testData[MAMA_MIDDLEWARE_SOLACE]     = "SOLACE";
>+    testData[MAMA_MIDDLEWARE_RAI]        = "rai";
>+    testData[MAMA_MIDDLEWARE_QPID]       = "QPID";
>+    testData[MAMA_MIDDLEWARE_EXEGY]      = "exegy";
>+    testData[MAMA_MIDDLEWARE_IBMWFO]     = "ibmwfo";
>+    testData[MAMA_MIDDLEWARE_ACTIV]      = "activ";
>+    testData[MAMA_MIDDLEWARE_TICK42RMDS] = "tick42rmds";
>+    testData[MAMA_MIDDLEWARE_UMS]        = "ums";
>+    testData[MAMA_MIDDLEWARE_VULCAN]     = "vulcan";
>+    testData[MAMA_MIDDLEWARE_UNKNOWN]    = "unknown";
>     MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); ++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaMiddleware middleware = (*itr).first;
>-       std::string    expected   = (*itr).second;
>-
>-       std::string actual = mamaMiddleware_convertToString (middleware);
>+       mamaMiddleware middleware = itr->first;
>+       const char*    name       = itr->second.c_str();
>
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       EXPECT_STREQ(name, mamaMiddleware_convertToString (middleware));
>+       EXPECT_EQ((char)middleware,
>(char)mamaMiddleware_convertFromString (name));
>     }
>-
>-    ASSERT_EQ (passed, true);
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertFromString)
>-{
>-    MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>-
>-    for (itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); itr++) {
>-
>-        mamaMiddleware expected    = (*itr).first;
>-        std::string    middleware  = (*itr).second;
>-
>-        mamaMiddleware actual      = mamaMiddleware_convertFromString
>(middleware.c_str());
>-
>-        EXPECT_EQ (actual, expected);
>-
>-        if (actual != expected)
>-            passed = false;
>-    }
>-
>-    ASSERT_EQ (passed, true);
>-}
>

________________________________________________________

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.
________________________________________________________




--
Lee Skillen

Vulcan Financial Technologies
51 Malone Road, Belfast, BT9 6RY

Web:     www.vulcanft.com 



--
Lee Skillen

Vulcan Financial Technologies
51 Malone Road, Belfast, BT9 6RY

Office:  +44 (0)28 95 817888
Mobile:  +44 (0)78 41 425152
Web:     www.vulcanft.com 

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


Lee Skillen <lskillen@...>
 

Hey Glenn,

Great to hear from you - We were thinking the same regarding the dynamic payload/middleware mappings and thought that this patch might be a good stepping stone towards that; a clean-up and centralisation without the necessity to make more drastic changes, if you will. ;-)  

If the identifiers were dynamic I suppose a mapping file of sorts would be checked in with the source code - Not sure if it would still be mama.properties format or not, but I guess it essentially has the same information that is hardcoded at the moment.  For example (illustrative, not really fully realised) :-

[middleware.avis]
id=3
name=Avis
description=Avis Middleware (Elvin Publish/Subscribe)
library=avis

[payload.avis]
id=A
name=AvisMsg
library=avismsg

A reasonable question is, without enumerations, what does the code in the bridges refer to?  I suppose it could do it based on any of the three id, name, or library.  If a bridge needs to refer to another bridge it would be the same thing - For example, picking the default payload for a middleware bridge.  Should that still be delegated to the bridge itself, which then refers to an id, name and/or library, or should it be loaded from the mapping file?  I guess backwards compat is a concern for code out there that still refers to enumerations for some bridge-specific hacks or something else.  I love APIs. :-)

Cheers,
Lee


On 25 February 2014 10:56, Glenn McClements <gmcclements@...> wrote:
Hi Lee,
At first glance the code looks good, and this is nice little feature with no adverse effects that I can think of. My only issue is that if we're tinkering with this area we should:
  • Go the whole hog and make payloads completely dynamic, removing the enum dependancy. Something like point OpenMAMA to a directory and it automatically loading all payloads in there, and query the payload on startup for it's identifier, not allowing two with the same identifier. There's some security issues with this I suppose.
  • Make it applicable to middleware (and any future kind of) bridges as well.

Happy to hear other peoples opinion on this. 

Cheers, 
Glenn 

From: Lee Skillen <lskillen@...>
Date: Thursday, 20 February 2014 13:42
To: Damian Maguire <DMaguire@...>

Cc: "openmama-dev@..." <openmama-dev@...>
Subject: Re: [Openmama-dev] [PATCH] MAMA: Add the capability to load payload libraries just-in-time (JIT).

Hey,

Just a quick update, it looks like our SMTP server caused the carriage returns - I've modified a version of git-send-email to include the capability to mime encode attachments as base64.  Tested and it carries through the patch in verbatim, so I shouldn't be sending any more patches that look like they came from a Windows machine. ;-)  I'll tidy up the git patch and submit it back to git-core next week since it seems useful (and may be useful to others here too).  Let me know if you need to me to re-submit the OpenMAMA patches I've already sent.

Thanks,
Lee



On 20 February 2014 17:08, Lee Skillen <lskillen@...> wrote:
Hey Damian,

Thanks, we're happy to finally be in a better position to start contributing.   First of all, just so that focus isn't purely on the JIT loading, I wanted to point out that this patch also has fixes for memory leaks, a reworking of the payload conversion unit tests, and a new style for the payload mappings (which we hope will be useful from a maintenance perspective to keep things centralised).

The new unit tests exercise the the string conversion capability and we ran the entire suite to ensure that nothing else has been broken.  Memory leaks are confirmed for presence before patch via valgrind and memory sanitizer and their removal is confirmed after.  We also utilise helgrind and thread sanitizer for threading issues in general, although not for this patch.

The JIT loading mechanism has been tested locally using a modified version of avis and our middleware.   Setup for a simple test would be to load a consuming application (e.g. mamaconsumerc) with any middleware that is payload agnostic, and then send a message utilising a payload that is different than the default payload for that middleware.  Pre-patch this would crash the application, but post-patch it'll load the payload just-in-time.

For testing the receiving bit is fine but none of the testtools applications have the capability to specify a different payload other than the default. So for the sake of demonstration I modified mamaproducerc to be capable of specifying payload on the command-line via -P <payload>.  The test looks like this:

Pre-patch:

$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v
Segmentation fault (core dumped)

Post-patch:
 
$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
[snip]
2014-02-20 16:57:20: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]
 
$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]
Starting to publish

Looking back at the mamaconsumer log we can see that it successfully loaded the vulcanmsg payload JIT :-

2014-02-20 16:58:09: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
Rate: 2 Latency(usec): Low: 13775       Average: 447183 High: 910969    Gaps: 1
Rate: 9 Latency(usec): Low: 34754       Average: 507466 High: 933766    Gaps: 0
Rate: 9 Latency(usec): Low: 56764       Average: 528090 High: 954780    Gaps: 0

Hope that looks good for you guys - Please feel free to experiment on the new branch.  We'd really like to get this code in to the mainline if at all possible, and we've got quite a lot more code to contribute too.

Apologies about the carriage returns in the patch file - I don't personally use Windows and we utilise git-format-patch and git-send-email for the patch formatting and sending.  The patch prior to sending doesn't have any carriage return characters in it, but I looked at one I pulled back and it does have 0x0d 0x0a ("\r\n") for the line endings.  So it seems that either SMTP or git-send-email was the culprit.  I'll have a look into preventing it but in the short-term `git apply --whitespace=fix` should sort it. :-(

Cracking start for submitting patches, eh. :-)

Cheers,
Lee




On 20 February 2014 14:36, Damian Maguire <DMaguire@...> wrote:
Cheers for this Lee, looks very interesting. From a quick look/preliminary
test the changes look solid, but given the size and scope of the change I
think the best thing to do is fire it into it's own branch for a while and
let people check it out for themselves. As such I'm going to stick it into
a feature-jit-payload branch on the main repo. Anyone who's interested can
then have a look and comment on the changes, and we can fix any bugs there
as required.

Regarding testing these changes, have you guys actually got this running
with any of the open source bridges, or do you have any sample testing you
can show us to demonstrate that it both works, and doesn't break the
existing code?

Just a quick note on the patch files you've attached - they seem to have
landed (to me at least) with DOS line endings, which messes with the git
apply step. It's not a big deal, since I can easily enough strip the line
endings, but probably something to watch out for with future submissions
(particularly if you touch any of the unfortunate files that already mix
and match line endings).

I'm also taking a look at the other patches you've sent through, so I'll
get back to you on those.

Cheers,

Damian



On 2/19/14 3:53 PM, "Lee Skillen" <lskillen@...> wrote:

>
>If a library hasn't been previously loaded then OpenMAMA will now attempt
>to load it by converting the payload identifier into a well-known string
>for the payload library.  This behaviour is useful if the payload isn't
>known ahead of time or if lazy loading is desired over enumeration of a
>existing set of libraries.  No changes made to existing API interfaces.
>
>The following functions have been added :-
>
>- mamaPayload_convertToLibString():
>  Maps a mamaPayloadType enumeration to the well-known library name.
>
>- mamaPayload_convertFromLibString():
>  Inverse of mamaPayload_convertToLibString.
>
>- mama_loadPayloadBridgeByType():
>  Takes a mamaPayloadType and utilises mamaPayload_convertToLibString()
>  to retrieve the library name for the payload, and then delegates this
>  to the existing mama_loadPayloadBridge() function.
>
>The string conversions are implemented as a linear search, primarily due
>to these functions not being on the hot path - Their use is prior to a
>library being loaded and (I would imagine) would be seldom used in other
>situations.
>
>Additional changes :-
>
>If msgPayload_destroyImpl has been exposed by the payload library then
>this is now utilised to cleanup the payload upon shutdown properly.  It
>was observed that each payload always leaked memory due to lacking the
>capability to destroy itself.  Individual payloads will still need to
>expose this - Backwards compatible as it won't be called otherwise.
>
>The MamaEnumTestsC set of unit tests have been rewritten to utilise the
>new conversion functionality.  The tests have also been renamed as such:
>
>- testPayloadToString -> testPayloadStringConversions
>- testPayloadFromString -> testPayloadStringConversions
>- testMiddlewareToString -> testMiddlewareStringConversions
>- testMiddlewareFromString -> testMiddlewareStringConversions
>
>Noticed that the ASSERT_EQ/EXPECT_EQ statements in gtest are sometimes
>used incorrectly, in that the expected/actual are reversed (it should be
>expected on the left and actual on the right).
>
>Todo :-
>
>Need to confirm well-known library names with all vendors - Will only be
>important if JIT loading is actually used, but the names should be made
>accurate anyway.
>
>Signed-off-by: Lee Skillen <lskillen@...>
>---
> mama/c_cpp/src/c/mama.c                            |  77 ++++++++--
> mama/c_cpp/src/c/mama/mama.h                       |  27 +++-
> mama/c_cpp/src/c/mama/msg.h                        |  30 ++++
> mama/c_cpp/src/c/msg.c                             | 163
>+++++++++++++++------
> mama/c_cpp/src/c/payloadbridge.h                   |   4 +
> .../src/gunittest/c/payloadmiddlewareidtest.cpp    | 152
>++++++++-----------
> 6 files changed, 304 insertions(+), 149 deletions(-)
>
>diff --git a/mama/c_cpp/src/c/mama.c b/mama/c_cpp/src/c/mama.c
>index 33a0768..3a7d7be 100644
>--- a/mama/c_cpp/src/c/mama.c
>+++ b/mama/c_cpp/src/c/mama.c
>@@ -28,6 +28,7 @@
> #include "wombat/strutils.h"
>
> #include <mama/mama.h>
>+#include <mama/msg.h>
> #include <mama/error.h>
> #include <mamainternal.h>
> #include <mama/version.h>
>@@ -168,6 +169,7 @@ typedef struct mamaImpl_
>     mamaPayloadBridge      myPayloads[MAMA_PAYLOAD_MAX];
>     LIB_HANDLE             myBridgeLibraries[MAMA_MIDDLEWARE_MAX];
>     LIB_HANDLE             myPayloadLibraries[MAMA_PAYLOAD_MAX];
>+    msgPayload_destroyImpl myPayloadDestroyFuncs[MAMA_PAYLOAD_MAX];
>     unsigned int           myRefCount;
>     wthread_static_mutex_t myLock;
> } mamaImpl;
>@@ -175,7 +177,7 @@ typedef struct mamaImpl_
> static mamaApplicationContext  appContext;
> static char mama_ver_string[256];
>
>-static mamaImpl gImpl = {{0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>+static mamaImpl gImpl = {{0}, {0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>
> /*
>*************************************************************************
>*/
> /* Private Function Prototypes. */
>@@ -624,10 +626,21 @@ mamaInternal_findBridge ()
> mamaPayloadBridge
> mamaInternal_findPayload (char id)
> {
>+    mamaPayloadBridge bridge = NULL;
>+
>     if ('\0' == id)
>         return NULL;
>
>-    return gImpl.myPayloads[(uint8_t)id];
>+    bridge = gImpl.myPayloads[(uint8_t)id];
>+
>+    if (NULL == bridge)
>+    {
>+        /* Dynamically load the payload bridge just-in-time, if it
>hasn't
>+         * already been loaded previously. */
>+        mama_loadPayloadBridgeByType (&bridge, (mamaPayloadType)id);
>+    }
>+
>+    return bridge;
> }
>
> mamaPayloadBridge
>@@ -1241,17 +1254,33 @@ mama_closeCount (unsigned int* count)
>
>         cleanupReservedFields();
>
>-         /* Look for a bridge for each of the payloads and close them */
>+        /* Look for a bridge for each of the payloads and close them */
>         for (payload = 0; payload != MAMA_PAYLOAD_MAX; ++payload)
>         {
>-              /* mamaPayloadBridgeImpl* impl = (mamaPayloadBridgeImpl*)
>-             * gImpl.myPayloads [(uint8_t)payload];*/
>-            gImpl.myPayloads[(uint8_t)payload] = NULL;
>-            if(gImpl.myPayloadLibraries[(uint8_t)payload])
>+            uint8_t                id   = (uint8_t)payload;
>+            LIB_HANDLE             lib  = gImpl.myPayloadLibraries[id];
>+            mamaPayloadBridgeImpl* impl = gImpl.myPayloads[id];
>+
>+            if (!lib)
>             {
>-                closeSharedLib
>(gImpl.myPayloadLibraries[(uint8_t)payload]);
>-                gImpl.myPayloadLibraries[(uint8_t)payload] = NULL;
>+                continue;
>+            }
>+
>+            if (impl)
>+            {
>+                msgPayload_destroyImpl destroyFunc =
>+                    gImpl.myPayloadDestroyFuncs[id];
>+
>+                if (destroyFunc)
>+                {
>+                    destroyFunc (impl);
>+                }
>+
>+                gImpl.myPayloads[id] = NULL;
>             }
>+
>+            closeSharedLib (lib);
>+            gImpl.myPayloadLibraries[id] = NULL;
>         }
>
>        gDefaultPayload = NULL;
>@@ -1827,18 +1856,29 @@ mama_setDefaultPayload (char id)
>
> mama_status
> mama_loadPayloadBridge (mamaPayloadBridge* impl,
>-                         const char*        payloadName)
>+                        const char*        payloadName)
> {
>     return mama_loadPayloadBridgeInternal (impl, payloadName);
> }
>+
>+mama_status
>+mama_loadPayloadBridgeByType (mamaPayloadBridge* impl,
>+                              mamaPayloadType    payload)
>+{
>+    const char* name = mamaPayload_convertToLibString (payload);
>+    return mama_loadPayloadBridgeInternal (impl, name);
>+}
>+
> mama_status
> mama_loadPayloadBridgeInternal  (mamaPayloadBridge* impl,
>                                  const char*        payloadName)
> {
>     char                    bridgeImplName  [256];
>     char                    initFuncName    [256];
>+    char                    destroyFuncName [256];
>     LIB_HANDLE              bridgeLib       = NULL;
>     msgPayload_createImpl   initFunc        = NULL;
>+    msgPayload_destroyImpl  destroyFunc     = NULL;
>     mama_status             status          = MAMA_STATUS_OK;
>     char                    payloadChar       ='\0';
>       void*                                   vp                              = NULL;
>@@ -1865,11 +1905,10 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>         return MAMA_STATUS_NO_BRIDGE_IMPL;
>     }
>
>-    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>-
>     /* Gives a warning - casting from void* to bridge_createImpl func */
>-      vp = loadLibFunc (bridgeLib, initFuncName);
>-      initFunc  = *(msgPayload_createImpl*) &vp;
>+    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>+    vp = loadLibFunc (bridgeLib, initFuncName);
>+    initFunc  = *(msgPayload_createImpl*) &vp;
>
>     if (!initFunc)
>     {
>@@ -1887,7 +1926,7 @@ mama_loadPayloadBridgeInternal  (mamaPayloadBridge*
>impl,
>
>     if (MAMA_STATUS_OK != (status = initFunc (impl, &payloadChar)))
>     {
>-       wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return status;
>     }
>@@ -1910,13 +1949,19 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>              "Payload bridge %s already loaded",
>              payloadName);
>
>-         wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return MAMA_STATUS_OK;
>     }
>
>+    /* Gives a warning - casting from void* to bridge_destroyImpl func */
>+    snprintf (destroyFuncName, 256, "%sPayload_destroyImpl",
>payloadName);
>+    vp = loadLibFunc (bridgeLib, destroyFuncName);
>+    destroyFunc = *(msgPayload_destroyImpl*) &vp;
>+
>     gImpl.myPayloads [(int)payloadChar] = *impl;
>     gImpl.myPayloadLibraries [(int)payloadChar] = bridgeLib;
>+    gImpl.myPayloadDestroyFuncs [(int)payloadChar] = destroyFunc;
>
>     if (!gDefaultPayload)
>     {
>diff --git a/mama/c_cpp/src/c/mama/mama.h b/mama/c_cpp/src/c/mama/mama.h
>index a574f36..07a9115 100644
>--- a/mama/c_cpp/src/c/mama/mama.h
>+++ b/mama/c_cpp/src/c/mama/mama.h
>@@ -133,11 +133,34 @@ extern "C"
>      */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadBridge (mamaBridge* impl, const char*  middleware);
>+    mama_loadBridge (mamaBridge* impl,
>+                     const char* middleware);
>
>+    /**
>+     * Load a payload bridge by its well-known library name string.
>E.g. "wmsg"
>+     * will load the WombatMsg payload, provided it exists on the system.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payloadName The well-known name of the payload library.
>+     */
>+    MAMAExpDLL
>+    extern mama_status
>+    mama_loadPayloadBridge (mamaPayloadBridge* bridge,
>+                            const char*        payloadName);
>+
>+    /**
>+     * Load a payload bridge by using the mamaPayloadType identifier.
>This
>+     * utilises mamaPayload_convertToLibString() in order to retrieve the
>+     * well-known library string for the payload library, and this is
>then
>+     * delegated to mama_loadPayloadBridge.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payload The payload enumeration identifier.
>+     */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadPayloadBridge (mamaPayloadBridge*  bridge,  const char*
>payloadName);
>+    mama_loadPayloadBridgeByType (mamaPayloadBridge* bridge,
>+                                  mamaPayloadType    payload);
>
>     /** Load the bridge specified by middleware string using the path
>specified by the user.
>      * If the bridge has already been loaded then the existing bridge
>instance
>diff --git a/mama/c_cpp/src/c/mama/msg.h b/mama/c_cpp/src/c/mama/msg.h
>index bf0d562..a9c3438 100644
>--- a/mama/c_cpp/src/c/mama/msg.h
>+++ b/mama/c_cpp/src/c/mama/msg.h
>@@ -74,6 +74,36 @@ extern const char*
> mamaPayload_convertToString (mamaPayloadType payloadType);
>
> /**
>+ * Convert a string to a mamaPayloadType.
>+ *
>+ * @param str The str to convert.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromString (const char* str);
>+
>+/**
>+ * Convert a mamaPayloadType value to a well-known library name string
>+ * for that payload.  E.g. MAMA_PAYLOAD_WOMBAT_MSG translates into
>+ * "wmsg" (of which the payload library would be libwmsgimpl.so on *nix).
>+ *
>+ * @param payloadType The payloadType to convert to a library name
>string.
>+ */
>+MAMAExpDLL
>+extern const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType);
>+
>+/**
>+ * Convert a well-known library name to a mamaPayloadType value
>+ * for that payload.  E.g. "wmsg" translates into
>MAMA_PAYLOAD_WOMBAT_MSG.
>+ *
>+ * @param str The library name string to convert to a mamaPayloadType.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str);
>+
>+/**
>  * Create a mamaMsg.
>  *
>  * @param msg The location of a mamaMsg where to store the result.
>diff --git a/mama/c_cpp/src/c/msg.c b/mama/c_cpp/src/c/msg.c
>index 286fcad..506c390 100644
>--- a/mama/c_cpp/src/c/msg.c
>+++ b/mama/c_cpp/src/c/msg.c
>@@ -41,13 +41,39 @@
>
> #include "wombat/wincompat.h"
>
>-
> #define MAX_SUBJECT 256
>
>-#define NOFID 0
>-
>-static const int INCLUDE_FIELD_NAME = (1 << 4);
>-static const int INCLUDE_FIELD_ID   = (1 << 2);
>+typedef struct PayloadTable
>+{
>+    mamaPayloadType mType;
>+    const char*     mName;
>+    const char*     mLibrary;
>+}
>+PayloadTable;
>+
>+static
>+const PayloadTable kPayloads [] =
>+{
>+    {MAMA_PAYLOAD_ACTIV,      "activ",      "activmsg"},
>+    {MAMA_PAYLOAD_AVIS,       "AVIS",       "avismsg"},
>+    {MAMA_PAYLOAD_EXEGY,      "EXEGY",      "exegymsg"},
>+    {MAMA_PAYLOAD_FAST,       "FAST",       "fastmsg"},
>+    {MAMA_PAYLOAD_IBMWFO,     "ibmwfo",     "ibmwfomsg"},
>+    {MAMA_PAYLOAD_INRUSH,     "INRUSH",     "inrushmsg"},
>+    {MAMA_PAYLOAD_QPID,       "QPID",       "qpidmsg"},
>+    {MAMA_PAYLOAD_RAI,        "rai",        "raimsg"},
>+    {MAMA_PAYLOAD_SOLACE,     "solacemsg",  "solacemsg"},
>+    {MAMA_PAYLOAD_TIBRV,      "TIBRV",      "tibrvmsg"},
>+    {MAMA_PAYLOAD_TICK42BLP,  "TICK42BLP",  "tick42blpmsg"},
>+    {MAMA_PAYLOAD_TICK42RMDS, "TICK42RMDS", "tick42rmdsmsg"},
>+    {MAMA_PAYLOAD_UMS,        "UMS",        "umsmsg"},
>+    {MAMA_PAYLOAD_V5,         "V5",         "wcachemsg"},
>+    {MAMA_PAYLOAD_VULCAN,     "Vulcan",     "vulcanmsg"},
>+    {MAMA_PAYLOAD_WOMBAT_MSG, "WombatMsg",  "wmsg"},
>+
>+    /* Linear search requires unknown to be at the end-of-table */
>+    {MAMA_PAYLOAD_UNKNOWN,    "unknown",    NULL}
>+};
>
> typedef struct mamaMsgIteratorImpl
> {
>@@ -106,6 +132,66 @@ typedef struct mamaMsgImpl_
> static mama_status
> mamaMsgImpl_destroyLastVectorMsg (mamaMsgImpl *impl);
>
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByType (mamaPayloadType payloadType)
>+{
>+    int k = 0;
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (kPayloads[k].mType == payloadType)
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByName (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mName))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByLibrary (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mLibrary))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
> /*=================================================================
>   = Public functions - defined in mama/msg.h
>   ================================================================*/
>@@ -166,7 +252,9 @@ mamaMsg_destroy (mamaMsg msg)
>     if (impl->mCurrentField)
>     {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>     }
>+
>     /*Destroy the reusable field object*/
>     if (impl->mCopy)
>     {
>@@ -366,44 +454,29 @@ mamaMsgImpl_getPayload (const mamaMsg msg,
>msgPayload* payload)
> const char*
> mamaPayload_convertToString (mamaPayloadType payloadType)
> {
>-    switch (payloadType)
>-    {
>-        case MAMA_PAYLOAD_SOLACE:
>-            return "solacemsg";
>-        case MAMA_PAYLOAD_V5:
>-            return "V5";
>-        case MAMA_PAYLOAD_AVIS:
>-            return "AVIS";
>-        case MAMA_PAYLOAD_TICK42BLP:
>-            return "TICK42BLP";
>-        case MAMA_PAYLOAD_FAST:
>-            return "FAST";
>-        case MAMA_PAYLOAD_RAI:
>-            return "rai";
>-        case MAMA_PAYLOAD_UMS:
>-            return "UMS";
>-        case MAMA_PAYLOAD_TICK42RMDS:
>-            return "TICK42RMDS";
>-        case MAMA_PAYLOAD_QPID:
>-            return "QPID";
>-        case MAMA_PAYLOAD_TIBRV:
>-            return "TIBRV";
>-        case MAMA_PAYLOAD_IBMWFO:
>-            return "ibmwfo";
>-        case MAMA_PAYLOAD_ACTIV:
>-            return "activ";
>-        case MAMA_PAYLOAD_VULCAN:
>-            return "Vulcan";
>-        case MAMA_PAYLOAD_WOMBAT_MSG:
>-            return "WombatMsg";
>-        case MAMA_PAYLOAD_EXEGY:
>-            return "EXEGY";
>-        case MAMA_PAYLOAD_INRUSH:
>-            return "INRUSH";
>-
>-        default:
>-            return "unknown";
>-    }
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mName;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByName (str);
>+    return table->mType;
>+}
>+
>+const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mLibrary;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByLibrary
>(str);
>+    return table->mType;
> }
>
> mama_status
>@@ -3481,7 +3554,11 @@ mama_status mamaMsgIterator_destroy
>(mamaMsgIterator iterator)
>     }
>
>     if (impl->mCurrentField)
>+    {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>+    }
>+
>     free (impl);
>     return MAMA_STATUS_OK;
> }
>diff --git a/mama/c_cpp/src/c/payloadbridge.h
>b/mama/c_cpp/src/c/payloadbridge.h
>index 703c1a9..6ad7108 100644
>--- a/mama/c_cpp/src/c/payloadbridge.h
>+++ b/mama/c_cpp/src/c/payloadbridge.h
>@@ -422,8 +422,12 @@ while(0)
>                       \
> typedef mama_status
> (*msgPayload_createImpl)      (mamaPayloadBridge* result, char*
>identifier);
>
>+typedef mama_status
>+(*msgPayload_destroyImpl)     (mamaPayloadBridge result);
>+
> typedef mamaPayloadType
> (*msgPayload_getType)         (void);
>+
> /*===================================================================
>  =              msgPayload bridge function pointers                 =
>  ====================================================================*/
>diff --git a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>index ed18a64..1e14227 100644
>--- a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>+++ b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>@@ -20,14 +20,9 @@ protected:
>
>    virtual void TearDown(void);
>
>-   typedef std::pair <mamaPayloadType, std::string> MamaPayloadPair;
>-   typedef std::map  <mamaPayloadType, std::string> MamaPayloadMapType;
>-
>-   typedef std::pair <mamaMiddleware,  std::string> MamaMiddlewarePair;
>-   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>-
>-   MamaPayloadMapType     payloadTestData;
>-   MamaMiddlewareMapType  middlewareTestData;
>+   typedef std::pair <std::string,     std::string>    PayloadNameLib;
>+   typedef std::map  <mamaPayloadType, PayloadNameLib>
>MamaPayloadMapType;
>+   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>
> private:
>    void CreateTestData();
>@@ -55,100 +50,81 @@ MamaEnumTestsC::TearDown()
> void
> MamaEnumTestsC::CreateTestData()
> {
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_V5,
>"V5"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_AVIS,
>"AVIS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42BLP,
>"TICK42BLP"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_FAST,
>"FAST"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_RAI,
>"rai"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UMS,
>"UMS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42RMDS,
>"TICK42RMDS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_QPID,
>"QPID"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TIBRV,
>"TIBRV"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_IBMWFO,
>"ibmwfo"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_ACTIV,
>"activ"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_VULCAN,
>"Vulcan"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_WOMBAT_MSG,
>"WombatMsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_EXEGY,
>"EXEGY"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_SOLACE,
>"solacemsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UNKNOWN,
>"unknown"));
>-
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_WMW,
>     "wmw"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_LBM,
>     "lbm"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TIBRV,     "tibrv"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_AVIS,
>     "AVIS"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42BLP, "tick42blp"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_SOLACE,    "SOLACE"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_RAI,
>     "rai"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_QPID,
>     "QPID"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_EXEGY,     "exegy"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_IBMWFO,    "ibmwfo"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_ACTIV,     "activ"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42RMDS, "tick42rmds"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_UMS,
>     "ums"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_VULCAN,    "vulcan"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_UNKNOWN,   "unknown"));
> }
>
>-
>-TEST_F (MamaEnumTestsC, testPayloadConvertToString)
>+TEST_F (MamaEnumTestsC, testPayloadStringConversions)
> {
>+    MamaPayloadMapType testData;
>+    testData[MAMA_PAYLOAD_ACTIV]      = PayloadNameLib ("activ",
>"activmsg");
>+    testData[MAMA_PAYLOAD_AVIS]       = PayloadNameLib ("AVIS",
>"avismsg");
>+    testData[MAMA_PAYLOAD_EXEGY]      = PayloadNameLib ("EXEGY",
>"exegymsg");
>+    testData[MAMA_PAYLOAD_FAST]       = PayloadNameLib ("FAST",
>"fastmsg");
>+    testData[MAMA_PAYLOAD_IBMWFO]     = PayloadNameLib ("ibmwfo",
>"ibmwfomsg");
>+    testData[MAMA_PAYLOAD_QPID]       = PayloadNameLib ("QPID",
>"qpidmsg");
>+    testData[MAMA_PAYLOAD_RAI]        = PayloadNameLib ("rai",
>"raimsg");
>+    testData[MAMA_PAYLOAD_SOLACE]     = PayloadNameLib ("solacemsg",
>"solacemsg");
>+    testData[MAMA_PAYLOAD_TIBRV]      = PayloadNameLib ("TIBRV",
>"tibrvmsg");
>+    testData[MAMA_PAYLOAD_TICK42BLP]  = PayloadNameLib ("TICK42BLP",
>"tick42blpmsg");
>+    testData[MAMA_PAYLOAD_TICK42RMDS] = PayloadNameLib ("TICK42RMDS",
>"tick42rmdsmsg");
>+    testData[MAMA_PAYLOAD_UMS]        = PayloadNameLib ("UMS",
>"umsmsg");
>+    testData[MAMA_PAYLOAD_VULCAN]     = PayloadNameLib ("Vulcan",
>"vulcanmsg");
>+    testData[MAMA_PAYLOAD_WOMBAT_MSG] = PayloadNameLib ("WombatMsg",
>"wmsg");
>+    testData[MAMA_PAYLOAD_V5]         = PayloadNameLib ("V5",
>"wcachemsg");
>+    testData[MAMA_PAYLOAD_UNKNOWN]    = PayloadNameLib ("unknown",
>"");
>+
>     MamaPayloadMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = payloadTestData.begin(); itr != payloadTestData.end();
>++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaPayloadType payload   = (*itr).first;
>-       std::string     expected  = (*itr).second;
>-
>-       std::string actual = mamaPayload_convertToString (payload);
>-
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       mamaPayloadType payload = itr->first;
>+       PayloadNameLib& namelib = itr->second;
>+       const char*     name    = namelib.first.c_str();
>+       const char*     library = namelib.second.c_str();
>+
>+       // Override for unknown, mama returns NULL for unknown library
>(so that
>+       // it prevents libraries from trying to load libmamaunknownmsg).
>+       if (MAMA_PAYLOAD_UNKNOWN == payload)
>+       {
>+           library = NULL;
>+       }
>+
>+       // Check name/payload conversion
>+       EXPECT_STREQ(name, mamaPayload_convertToString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromString
>(name));
>+
>+       // Check library/payload conversion
>+       EXPECT_STREQ(library, mamaPayload_convertToLibString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromLibString
>(library));
>     }
>-
>-    ASSERT_EQ( passed, true );
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertToString)
>+TEST_F (MamaEnumTestsC, testMiddlewareStringConversions)
> {
>+    MamaMiddlewareMapType testData;
>+    testData[MAMA_MIDDLEWARE_WMW]        = "wmw";
>+    testData[MAMA_MIDDLEWARE_LBM]        = "lbm";
>+    testData[MAMA_MIDDLEWARE_TIBRV]      = "tibrv";
>+    testData[MAMA_MIDDLEWARE_AVIS]       = "AVIS";
>+    testData[MAMA_MIDDLEWARE_TICK42BLP]  = "tick42blp";
>+    testData[MAMA_MIDDLEWARE_SOLACE]     = "SOLACE";
>+    testData[MAMA_MIDDLEWARE_RAI]        = "rai";
>+    testData[MAMA_MIDDLEWARE_QPID]       = "QPID";
>+    testData[MAMA_MIDDLEWARE_EXEGY]      = "exegy";
>+    testData[MAMA_MIDDLEWARE_IBMWFO]     = "ibmwfo";
>+    testData[MAMA_MIDDLEWARE_ACTIV]      = "activ";
>+    testData[MAMA_MIDDLEWARE_TICK42RMDS] = "tick42rmds";
>+    testData[MAMA_MIDDLEWARE_UMS]        = "ums";
>+    testData[MAMA_MIDDLEWARE_VULCAN]     = "vulcan";
>+    testData[MAMA_MIDDLEWARE_UNKNOWN]    = "unknown";
>     MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); ++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaMiddleware middleware = (*itr).first;
>-       std::string    expected   = (*itr).second;
>-
>-       std::string actual = mamaMiddleware_convertToString (middleware);
>+       mamaMiddleware middleware = itr->first;
>+       const char*    name       = itr->second.c_str();
>
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       EXPECT_STREQ(name, mamaMiddleware_convertToString (middleware));
>+       EXPECT_EQ((char)middleware,
>(char)mamaMiddleware_convertFromString (name));
>     }
>-
>-    ASSERT_EQ (passed, true);
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertFromString)
>-{
>-    MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>-
>-    for (itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); itr++) {
>-
>-        mamaMiddleware expected    = (*itr).first;
>-        std::string    middleware  = (*itr).second;
>-
>-        mamaMiddleware actual      = mamaMiddleware_convertFromString
>(middleware.c_str());
>-
>-        EXPECT_EQ (actual, expected);
>-
>-        if (actual != expected)
>-            passed = false;
>-    }
>-
>-    ASSERT_EQ (passed, true);
>-}
>

________________________________________________________

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.
________________________________________________________




--
Lee Skillen

Vulcan Financial Technologies
51 Malone Road, Belfast, BT9 6RY

Web:     www.vulcanft.com 



--
Lee Skillen

Vulcan Financial Technologies
51 Malone Road, Belfast, BT9 6RY

Web:     www.vulcanft.com 

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.



--
Lee Skillen

Vulcan Financial Technologies
51 Malone Road, Belfast, BT9 6RY

Office:  +44 (0)28 95 817888
Mobile:  +44 (0)78 41 425152
Web:     www.vulcanft.com 


Sanjeev Wahi <wahi@...>
 

One approach could be, to keep it consistent with existing approaches maybe we can have a “PAYLOAD_PATH” an environment variable pointing to a directories path containing payload info. *(Just like WOMBAT_PATH is being searched for mama.properties)

 

-Regards,

Sanjeev

 

 

Copyright © 2014 Realtime Data Services, LLC. All rights reserved. | www.RtdsLLC.com

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Glenn McClements
Sent: Tuesday, February 25, 2014 5:57 AM
To: Lee Skillen; Damian Maguire
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] [PATCH] MAMA: Add the capability to load payload libraries just-in-time (JIT).

 

Hi Lee,

At first glance the code looks good, and this is nice little feature with no adverse effects that I can think of. My only issue is that if we're tinkering with this area we should:

  • Go the whole hog and make payloads completely dynamic, removing the enum dependancy. Something like point OpenMAMA to a directory and it automatically loading all payloads in there, and query the payload on startup for it's identifier, not allowing two with the same identifier. There's some security issues with this I suppose.
  • Make it applicable to middleware (and any future kind of) bridges as well.

 

Happy to hear other peoples opinion on this. 

 

Cheers, 

Glenn 

 

From: Lee Skillen <lskillen@...>
Date: Thursday, 20 February 2014 13:42
To: Damian Maguire <DMaguire@...>
Cc: "openmama-dev@..." <openmama-dev@...>
Subject: Re: [Openmama-dev] [PATCH] MAMA: Add the capability to load payload libraries just-in-time (JIT).

 

Hey,

 

Just a quick update, it looks like our SMTP server caused the carriage returns - I've modified a version of git-send-email to include the capability to mime encode attachments as base64.  Tested and it carries through the patch in verbatim, so I shouldn't be sending any more patches that look like they came from a Windows machine. ;-)  I'll tidy up the git patch and submit it back to git-core next week since it seems useful (and may be useful to others here too).  Let me know if you need to me to re-submit the OpenMAMA patches I've already sent.

 

Thanks,
Lee

 

 

On 20 February 2014 17:08, Lee Skillen <lskillen@...> wrote:

Hey Damian,

 

Thanks, we're happy to finally be in a better position to start contributing.   First of all, just so that focus isn't purely on the JIT loading, I wanted to point out that this patch also has fixes for memory leaks, a reworking of the payload conversion unit tests, and a new style for the payload mappings (which we hope will be useful from a maintenance perspective to keep things centralised).

 

The new unit tests exercise the the string conversion capability and we ran the entire suite to ensure that nothing else has been broken.  Memory leaks are confirmed for presence before patch via valgrind and memory sanitizer and their removal is confirmed after.  We also utilise helgrind and thread sanitizer for threading issues in general, although not for this patch.

 

The JIT loading mechanism has been tested locally using a modified version of avis and our middleware.   Setup for a simple test would be to load a consuming application (e.g. mamaconsumerc) with any middleware that is payload agnostic, and then send a message utilising a payload that is different than the default payload for that middleware.  Pre-patch this would crash the application, but post-patch it'll load the payload just-in-time.

 

For testing the receiving bit is fine but none of the testtools applications have the capability to specify a different payload other than the default. So for the sake of demonstration I modified mamaproducerc to be capable of specifying payload on the command-line via -P <payload>.  The test looks like this:


Pre-patch:

 

$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v

Segmentation fault (core dumped)

 

Post-patch:

 

$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
[snip]
2014-02-20 16:57:20: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]

 

$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]
Starting to publish

 

Looking back at the mamaconsumer log we can see that it successfully loaded the vulcanmsg payload JIT :-

 

2014-02-20 16:58:09: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
Rate: 2 Latency(usec): Low: 13775       Average: 447183 High: 910969    Gaps: 1
Rate: 9 Latency(usec): Low: 34754       Average: 507466 High: 933766    Gaps: 0
Rate: 9 Latency(usec): Low: 56764       Average: 528090 High: 954780    Gaps: 0

 

Hope that looks good for you guys - Please feel free to experiment on the new branch.  We'd really like to get this code in to the mainline if at all possible, and we've got quite a lot more code to contribute too.

 

Apologies about the carriage returns in the patch file - I don't personally use Windows and we utilise git-format-patch and git-send-email for the patch formatting and sending.  The patch prior to sending doesn't have any carriage return characters in it, but I looked at one I pulled back and it does have 0x0d 0x0a ("\r\n") for the line endings.  So it seems that either SMTP or git-send-email was the culprit.  I'll have a look into preventing it but in the short-term `git apply --whitespace=fix` should sort it. :-(

 

Cracking start for submitting patches, eh. :-)

 

Cheers,

Lee

 

 

 

On 20 February 2014 14:36, Damian Maguire <DMaguire@...> wrote:

Cheers for this Lee, looks very interesting. From a quick look/preliminary
test the changes look solid, but given the size and scope of the change I
think the best thing to do is fire it into it's own branch for a while and
let people check it out for themselves. As such I'm going to stick it into
a feature-jit-payload branch on the main repo. Anyone who's interested can
then have a look and comment on the changes, and we can fix any bugs there
as required.

Regarding testing these changes, have you guys actually got this running
with any of the open source bridges, or do you have any sample testing you
can show us to demonstrate that it both works, and doesn't break the
existing code?

Just a quick note on the patch files you've attached - they seem to have
landed (to me at least) with DOS line endings, which messes with the git
apply step. It's not a big deal, since I can easily enough strip the line
endings, but probably something to watch out for with future submissions
(particularly if you touch any of the unfortunate files that already mix
and match line endings).

I'm also taking a look at the other patches you've sent through, so I'll
get back to you on those.

Cheers,

Damian




On 2/19/14 3:53 PM, "Lee Skillen" <lskillen@...> wrote:

>
>If a library hasn't been previously loaded then OpenMAMA will now attempt
>to load it by converting the payload identifier into a well-known string
>for the payload library.  This behaviour is useful if the payload isn't
>known ahead of time or if lazy loading is desired over enumeration of a
>existing set of libraries.  No changes made to existing API interfaces.
>
>The following functions have been added :-
>
>- mamaPayload_convertToLibString():
>  Maps a mamaPayloadType enumeration to the well-known library name.
>
>- mamaPayload_convertFromLibString():
>  Inverse of mamaPayload_convertToLibString.
>
>- mama_loadPayloadBridgeByType():
>  Takes a mamaPayloadType and utilises mamaPayload_convertToLibString()
>  to retrieve the library name for the payload, and then delegates this
>  to the existing mama_loadPayloadBridge() function.
>
>The string conversions are implemented as a linear search, primarily due
>to these functions not being on the hot path - Their use is prior to a
>library being loaded and (I would imagine) would be seldom used in other
>situations.
>
>Additional changes :-
>
>If msgPayload_destroyImpl has been exposed by the payload library then
>this is now utilised to cleanup the payload upon shutdown properly.  It
>was observed that each payload always leaked memory due to lacking the
>capability to destroy itself.  Individual payloads will still need to
>expose this - Backwards compatible as it won't be called otherwise.
>
>The MamaEnumTestsC set of unit tests have been rewritten to utilise the
>new conversion functionality.  The tests have also been renamed as such:
>
>- testPayloadToString -> testPayloadStringConversions
>- testPayloadFromString -> testPayloadStringConversions
>- testMiddlewareToString -> testMiddlewareStringConversions
>- testMiddlewareFromString -> testMiddlewareStringConversions
>
>Noticed that the ASSERT_EQ/EXPECT_EQ statements in gtest are sometimes
>used incorrectly, in that the expected/actual are reversed (it should be
>expected on the left and actual on the right).
>
>Todo :-
>
>Need to confirm well-known library names with all vendors - Will only be
>important if JIT loading is actually used, but the names should be made
>accurate anyway.
>
>Signed-off-by: Lee Skillen <lskillen@...>
>---
> mama/c_cpp/src/c/mama.c                            |  77 ++++++++--
> mama/c_cpp/src/c/mama/mama.h                       |  27 +++-
> mama/c_cpp/src/c/mama/msg.h                        |  30 ++++
> mama/c_cpp/src/c/msg.c                             | 163
>+++++++++++++++------
> mama/c_cpp/src/c/payloadbridge.h                   |   4 +
> .../src/gunittest/c/payloadmiddlewareidtest.cpp    | 152
>++++++++-----------
> 6 files changed, 304 insertions(+), 149 deletions(-)
>
>diff --git a/mama/c_cpp/src/c/mama.c b/mama/c_cpp/src/c/mama.c
>index 33a0768..3a7d7be 100644
>--- a/mama/c_cpp/src/c/mama.c
>+++ b/mama/c_cpp/src/c/mama.c
>@@ -28,6 +28,7 @@
> #include "wombat/strutils.h"
>
> #include <mama/mama.h>
>+#include <mama/msg.h>
> #include <mama/error.h>
> #include <mamainternal.h>
> #include <mama/version.h>
>@@ -168,6 +169,7 @@ typedef struct mamaImpl_
>     mamaPayloadBridge      myPayloads[MAMA_PAYLOAD_MAX];
>     LIB_HANDLE             myBridgeLibraries[MAMA_MIDDLEWARE_MAX];
>     LIB_HANDLE             myPayloadLibraries[MAMA_PAYLOAD_MAX];
>+    msgPayload_destroyImpl myPayloadDestroyFuncs[MAMA_PAYLOAD_MAX];
>     unsigned int           myRefCount;
>     wthread_static_mutex_t myLock;
> } mamaImpl;
>@@ -175,7 +177,7 @@ typedef struct mamaImpl_
> static mamaApplicationContext  appContext;
> static char mama_ver_string[256];
>
>-static mamaImpl gImpl = {{0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>+static mamaImpl gImpl = {{0}, {0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>
> /*
>*************************************************************************
>*/
> /* Private Function Prototypes. */
>@@ -624,10 +626,21 @@ mamaInternal_findBridge ()
> mamaPayloadBridge
> mamaInternal_findPayload (char id)
> {
>+    mamaPayloadBridge bridge = NULL;
>+
>     if ('\0' == id)
>         return NULL;
>
>-    return gImpl.myPayloads[(uint8_t)id];
>+    bridge = gImpl.myPayloads[(uint8_t)id];
>+
>+    if (NULL == bridge)
>+    {
>+        /* Dynamically load the payload bridge just-in-time, if it
>hasn't
>+         * already been loaded previously. */
>+        mama_loadPayloadBridgeByType (&bridge, (mamaPayloadType)id);
>+    }
>+
>+    return bridge;
> }
>
> mamaPayloadBridge
>@@ -1241,17 +1254,33 @@ mama_closeCount (unsigned int* count)
>
>         cleanupReservedFields();
>
>-         /* Look for a bridge for each of the payloads and close them */
>+        /* Look for a bridge for each of the payloads and close them */
>         for (payload = 0; payload != MAMA_PAYLOAD_MAX; ++payload)
>         {
>-              /* mamaPayloadBridgeImpl* impl = (mamaPayloadBridgeImpl*)
>-             * gImpl.myPayloads [(uint8_t)payload];*/
>-            gImpl.myPayloads[(uint8_t)payload] = NULL;
>-            if(gImpl.myPayloadLibraries[(uint8_t)payload])
>+            uint8_t                id   = (uint8_t)payload;
>+            LIB_HANDLE             lib  = gImpl.myPayloadLibraries[id];
>+            mamaPayloadBridgeImpl* impl = gImpl.myPayloads[id];
>+
>+            if (!lib)
>             {
>-                closeSharedLib
>(gImpl.myPayloadLibraries[(uint8_t)payload]);
>-                gImpl.myPayloadLibraries[(uint8_t)payload] = NULL;
>+                continue;
>+            }
>+
>+            if (impl)
>+            {
>+                msgPayload_destroyImpl destroyFunc =
>+                    gImpl.myPayloadDestroyFuncs[id];
>+
>+                if (destroyFunc)
>+                {
>+                    destroyFunc (impl);
>+                }
>+
>+                gImpl.myPayloads[id] = NULL;
>             }
>+
>+            closeSharedLib (lib);
>+            gImpl.myPayloadLibraries[id] = NULL;
>         }
>
>        gDefaultPayload = NULL;
>@@ -1827,18 +1856,29 @@ mama_setDefaultPayload (char id)
>
> mama_status
> mama_loadPayloadBridge (mamaPayloadBridge* impl,
>-                         const char*        payloadName)
>+                        const char*        payloadName)
> {
>     return mama_loadPayloadBridgeInternal (impl, payloadName);
> }
>+
>+mama_status
>+mama_loadPayloadBridgeByType (mamaPayloadBridge* impl,
>+                              mamaPayloadType    payload)
>+{
>+    const char* name = mamaPayload_convertToLibString (payload);
>+    return mama_loadPayloadBridgeInternal (impl, name);
>+}
>+
> mama_status
> mama_loadPayloadBridgeInternal  (mamaPayloadBridge* impl,
>                                  const char*        payloadName)
> {
>     char                    bridgeImplName  [256];
>     char                    initFuncName    [256];
>+    char                    destroyFuncName [256];
>     LIB_HANDLE              bridgeLib       = NULL;
>     msgPayload_createImpl   initFunc        = NULL;
>+    msgPayload_destroyImpl  destroyFunc     = NULL;
>     mama_status             status          = MAMA_STATUS_OK;
>     char                    payloadChar       ='\0';
>       void*                                   vp                              = NULL;
>@@ -1865,11 +1905,10 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>         return MAMA_STATUS_NO_BRIDGE_IMPL;
>     }
>
>-    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>-
>     /* Gives a warning - casting from void* to bridge_createImpl func */
>-      vp = loadLibFunc (bridgeLib, initFuncName);
>-      initFunc  = *(msgPayload_createImpl*) &vp;
>+    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>+    vp = loadLibFunc (bridgeLib, initFuncName);
>+    initFunc  = *(msgPayload_createImpl*) &vp;
>
>     if (!initFunc)
>     {
>@@ -1887,7 +1926,7 @@ mama_loadPayloadBridgeInternal  (mamaPayloadBridge*
>impl,
>
>     if (MAMA_STATUS_OK != (status = initFunc (impl, &payloadChar)))
>     {
>-       wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return status;
>     }
>@@ -1910,13 +1949,19 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>              "Payload bridge %s already loaded",
>              payloadName);
>
>-         wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return MAMA_STATUS_OK;
>     }
>
>+    /* Gives a warning - casting from void* to bridge_destroyImpl func */
>+    snprintf (destroyFuncName, 256, "%sPayload_destroyImpl",
>payloadName);
>+    vp = loadLibFunc (bridgeLib, destroyFuncName);
>+    destroyFunc = *(msgPayload_destroyImpl*) &vp;
>+
>     gImpl.myPayloads [(int)payloadChar] = *impl;
>     gImpl.myPayloadLibraries [(int)payloadChar] = bridgeLib;
>+    gImpl.myPayloadDestroyFuncs [(int)payloadChar] = destroyFunc;
>
>     if (!gDefaultPayload)
>     {
>diff --git a/mama/c_cpp/src/c/mama/mama.h b/mama/c_cpp/src/c/mama/mama.h
>index a574f36..07a9115 100644
>--- a/mama/c_cpp/src/c/mama/mama.h
>+++ b/mama/c_cpp/src/c/mama/mama.h
>@@ -133,11 +133,34 @@ extern "C"
>      */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadBridge (mamaBridge* impl, const char*  middleware);
>+    mama_loadBridge (mamaBridge* impl,
>+                     const char* middleware);
>
>+    /**
>+     * Load a payload bridge by its well-known library name string.
>E.g. "wmsg"
>+     * will load the WombatMsg payload, provided it exists on the system.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payloadName The well-known name of the payload library.
>+     */
>+    MAMAExpDLL
>+    extern mama_status
>+    mama_loadPayloadBridge (mamaPayloadBridge* bridge,
>+                            const char*        payloadName);
>+
>+    /**
>+     * Load a payload bridge by using the mamaPayloadType identifier.
>This
>+     * utilises mamaPayload_convertToLibString() in order to retrieve the
>+     * well-known library string for the payload library, and this is
>then
>+     * delegated to mama_loadPayloadBridge.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payload The payload enumeration identifier.
>+     */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadPayloadBridge (mamaPayloadBridge*  bridge,  const char*
>payloadName);
>+    mama_loadPayloadBridgeByType (mamaPayloadBridge* bridge,
>+                                  mamaPayloadType    payload);
>
>     /** Load the bridge specified by middleware string using the path
>specified by the user.
>      * If the bridge has already been loaded then the existing bridge
>instance
>diff --git a/mama/c_cpp/src/c/mama/msg.h b/mama/c_cpp/src/c/mama/msg.h
>index bf0d562..a9c3438 100644
>--- a/mama/c_cpp/src/c/mama/msg.h
>+++ b/mama/c_cpp/src/c/mama/msg.h
>@@ -74,6 +74,36 @@ extern const char*
> mamaPayload_convertToString (mamaPayloadType payloadType);
>
> /**
>+ * Convert a string to a mamaPayloadType.
>+ *
>+ * @param str The str to convert.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromString (const char* str);
>+
>+/**
>+ * Convert a mamaPayloadType value to a well-known library name string
>+ * for that payload.  E.g. MAMA_PAYLOAD_WOMBAT_MSG translates into
>+ * "wmsg" (of which the payload library would be libwmsgimpl.so on *nix).
>+ *
>+ * @param payloadType The payloadType to convert to a library name
>string.
>+ */
>+MAMAExpDLL
>+extern const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType);
>+
>+/**
>+ * Convert a well-known library name to a mamaPayloadType value
>+ * for that payload.  E.g. "wmsg" translates into
>MAMA_PAYLOAD_WOMBAT_MSG.
>+ *
>+ * @param str The library name string to convert to a mamaPayloadType.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str);
>+
>+/**
>  * Create a mamaMsg.
>  *
>  * @param msg The location of a mamaMsg where to store the result.
>diff --git a/mama/c_cpp/src/c/msg.c b/mama/c_cpp/src/c/msg.c
>index 286fcad..506c390 100644
>--- a/mama/c_cpp/src/c/msg.c
>+++ b/mama/c_cpp/src/c/msg.c
>@@ -41,13 +41,39 @@
>
> #include "wombat/wincompat.h"
>
>-
> #define MAX_SUBJECT 256
>
>-#define NOFID 0
>-
>-static const int INCLUDE_FIELD_NAME = (1 << 4);
>-static const int INCLUDE_FIELD_ID   = (1 << 2);
>+typedef struct PayloadTable
>+{
>+    mamaPayloadType mType;
>+    const char*     mName;
>+    const char*     mLibrary;
>+}
>+PayloadTable;
>+
>+static
>+const PayloadTable kPayloads [] =
>+{
>+    {MAMA_PAYLOAD_ACTIV,      "activ",      "activmsg"},
>+    {MAMA_PAYLOAD_AVIS,       "AVIS",       "avismsg"},
>+    {MAMA_PAYLOAD_EXEGY,      "EXEGY",      "exegymsg"},
>+    {MAMA_PAYLOAD_FAST,       "FAST",       "fastmsg"},
>+    {MAMA_PAYLOAD_IBMWFO,     "ibmwfo",     "ibmwfomsg"},
>+    {MAMA_PAYLOAD_INRUSH,     "INRUSH",     "inrushmsg"},
>+    {MAMA_PAYLOAD_QPID,       "QPID",       "qpidmsg"},
>+    {MAMA_PAYLOAD_RAI,        "rai",        "raimsg"},
>+    {MAMA_PAYLOAD_SOLACE,     "solacemsg",  "solacemsg"},
>+    {MAMA_PAYLOAD_TIBRV,      "TIBRV",      "tibrvmsg"},
>+    {MAMA_PAYLOAD_TICK42BLP,  "TICK42BLP",  "tick42blpmsg"},
>+    {MAMA_PAYLOAD_TICK42RMDS, "TICK42RMDS", "tick42rmdsmsg"},
>+    {MAMA_PAYLOAD_UMS,        "UMS",        "umsmsg"},
>+    {MAMA_PAYLOAD_V5,         "V5",         "wcachemsg"},
>+    {MAMA_PAYLOAD_VULCAN,     "Vulcan",     "vulcanmsg"},
>+    {MAMA_PAYLOAD_WOMBAT_MSG, "WombatMsg",  "wmsg"},
>+
>+    /* Linear search requires unknown to be at the end-of-table */
>+    {MAMA_PAYLOAD_UNKNOWN,    "unknown",    NULL}
>+};
>
> typedef struct mamaMsgIteratorImpl
> {
>@@ -106,6 +132,66 @@ typedef struct mamaMsgImpl_
> static mama_status
> mamaMsgImpl_destroyLastVectorMsg (mamaMsgImpl *impl);
>
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByType (mamaPayloadType payloadType)
>+{
>+    int k = 0;
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (kPayloads[k].mType == payloadType)
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByName (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mName))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByLibrary (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mLibrary))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
> /*=================================================================
>   = Public functions - defined in mama/msg.h
>   ================================================================*/
>@@ -166,7 +252,9 @@ mamaMsg_destroy (mamaMsg msg)
>     if (impl->mCurrentField)
>     {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>     }
>+
>     /*Destroy the reusable field object*/
>     if (impl->mCopy)
>     {
>@@ -366,44 +454,29 @@ mamaMsgImpl_getPayload (const mamaMsg msg,
>msgPayload* payload)
> const char*
> mamaPayload_convertToString (mamaPayloadType payloadType)
> {
>-    switch (payloadType)
>-    {
>-        case MAMA_PAYLOAD_SOLACE:
>-            return "solacemsg";
>-        case MAMA_PAYLOAD_V5:
>-            return "V5";
>-        case MAMA_PAYLOAD_AVIS:
>-            return "AVIS";
>-        case MAMA_PAYLOAD_TICK42BLP:
>-            return "TICK42BLP";
>-        case MAMA_PAYLOAD_FAST:
>-            return "FAST";
>-        case MAMA_PAYLOAD_RAI:
>-            return "rai";
>-        case MAMA_PAYLOAD_UMS:
>-            return "UMS";
>-        case MAMA_PAYLOAD_TICK42RMDS:
>-            return "TICK42RMDS";
>-        case MAMA_PAYLOAD_QPID:
>-            return "QPID";
>-        case MAMA_PAYLOAD_TIBRV:
>-            return "TIBRV";
>-        case MAMA_PAYLOAD_IBMWFO:
>-            return "ibmwfo";
>-        case MAMA_PAYLOAD_ACTIV:
>-            return "activ";
>-        case MAMA_PAYLOAD_VULCAN:
>-            return "Vulcan";
>-        case MAMA_PAYLOAD_WOMBAT_MSG:
>-            return "WombatMsg";
>-        case MAMA_PAYLOAD_EXEGY:
>-            return "EXEGY";
>-        case MAMA_PAYLOAD_INRUSH:
>-            return "INRUSH";
>-
>-        default:
>-            return "unknown";
>-    }
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mName;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByName (str);
>+    return table->mType;
>+}
>+
>+const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mLibrary;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByLibrary
>(str);
>+    return table->mType;
> }
>
> mama_status
>@@ -3481,7 +3554,11 @@ mama_status mamaMsgIterator_destroy
>(mamaMsgIterator iterator)
>     }
>
>     if (impl->mCurrentField)
>+    {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>+    }
>+
>     free (impl);
>     return MAMA_STATUS_OK;
> }
>diff --git a/mama/c_cpp/src/c/payloadbridge.h
>b/mama/c_cpp/src/c/payloadbridge.h
>index 703c1a9..6ad7108 100644
>--- a/mama/c_cpp/src/c/payloadbridge.h
>+++ b/mama/c_cpp/src/c/payloadbridge.h
>@@ -422,8 +422,12 @@ while(0)
>                       \
> typedef mama_status
> (*msgPayload_createImpl)      (mamaPayloadBridge* result, char*
>identifier);
>
>+typedef mama_status
>+(*msgPayload_destroyImpl)     (mamaPayloadBridge result);
>+
> typedef mamaPayloadType
> (*msgPayload_getType)         (void);
>+
> /*===================================================================
>  =              msgPayload bridge function pointers                 =
>  ====================================================================*/
>diff --git a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>index ed18a64..1e14227 100644
>--- a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>+++ b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>@@ -20,14 +20,9 @@ protected:
>
>    virtual void TearDown(void);
>
>-   typedef std::pair <mamaPayloadType, std::string> MamaPayloadPair;
>-   typedef std::map  <mamaPayloadType, std::string> MamaPayloadMapType;
>-
>-   typedef std::pair <mamaMiddleware,  std::string> MamaMiddlewarePair;
>-   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>-
>-   MamaPayloadMapType     payloadTestData;
>-   MamaMiddlewareMapType  middlewareTestData;
>+   typedef std::pair <std::string,     std::string>    PayloadNameLib;
>+   typedef std::map  <mamaPayloadType, PayloadNameLib>
>MamaPayloadMapType;
>+   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>
> private:
>    void CreateTestData();
>@@ -55,100 +50,81 @@ MamaEnumTestsC::TearDown()
> void
> MamaEnumTestsC::CreateTestData()
> {
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_V5,
>"V5"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_AVIS,
>"AVIS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42BLP,
>"TICK42BLP"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_FAST,
>"FAST"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_RAI,
>"rai"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UMS,
>"UMS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42RMDS,
>"TICK42RMDS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_QPID,
>"QPID"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TIBRV,
>"TIBRV"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_IBMWFO,
>"ibmwfo"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_ACTIV,
>"activ"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_VULCAN,
>"Vulcan"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_WOMBAT_MSG,
>"WombatMsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_EXEGY,
>"EXEGY"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_SOLACE,
>"solacemsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UNKNOWN,
>"unknown"));
>-
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_WMW,
>     "wmw"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_LBM,
>     "lbm"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TIBRV,     "tibrv"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_AVIS,
>     "AVIS"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42BLP, "tick42blp"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_SOLACE,    "SOLACE"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_RAI,
>     "rai"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_QPID,
>     "QPID"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_EXEGY,     "exegy"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_IBMWFO,    "ibmwfo"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_ACTIV,     "activ"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42RMDS, "tick42rmds"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_UMS,
>     "ums"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_VULCAN,    "vulcan"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_UNKNOWN,   "unknown"));
> }
>
>-
>-TEST_F (MamaEnumTestsC, testPayloadConvertToString)
>+TEST_F (MamaEnumTestsC, testPayloadStringConversions)
> {
>+    MamaPayloadMapType testData;
>+    testData[MAMA_PAYLOAD_ACTIV]      = PayloadNameLib ("activ",
>"activmsg");
>+    testData[MAMA_PAYLOAD_AVIS]       = PayloadNameLib ("AVIS",
>"avismsg");
>+    testData[MAMA_PAYLOAD_EXEGY]      = PayloadNameLib ("EXEGY",
>"exegymsg");
>+    testData[MAMA_PAYLOAD_FAST]       = PayloadNameLib ("FAST",
>"fastmsg");
>+    testData[MAMA_PAYLOAD_IBMWFO]     = PayloadNameLib ("ibmwfo",
>"ibmwfomsg");
>+    testData[MAMA_PAYLOAD_QPID]       = PayloadNameLib ("QPID",
>"qpidmsg");
>+    testData[MAMA_PAYLOAD_RAI]        = PayloadNameLib ("rai",
>"raimsg");
>+    testData[MAMA_PAYLOAD_SOLACE]     = PayloadNameLib ("solacemsg",
>"solacemsg");
>+    testData[MAMA_PAYLOAD_TIBRV]      = PayloadNameLib ("TIBRV",
>"tibrvmsg");
>+    testData[MAMA_PAYLOAD_TICK42BLP]  = PayloadNameLib ("TICK42BLP",
>"tick42blpmsg");
>+    testData[MAMA_PAYLOAD_TICK42RMDS] = PayloadNameLib ("TICK42RMDS",
>"tick42rmdsmsg");
>+    testData[MAMA_PAYLOAD_UMS]        = PayloadNameLib ("UMS",
>"umsmsg");
>+    testData[MAMA_PAYLOAD_VULCAN]     = PayloadNameLib ("Vulcan",
>"vulcanmsg");
>+    testData[MAMA_PAYLOAD_WOMBAT_MSG] = PayloadNameLib ("WombatMsg",
>"wmsg");
>+    testData[MAMA_PAYLOAD_V5]         = PayloadNameLib ("V5",
>"wcachemsg");
>+    testData[MAMA_PAYLOAD_UNKNOWN]    = PayloadNameLib ("unknown",
>"");
>+
>     MamaPayloadMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = payloadTestData.begin(); itr != payloadTestData.end();
>++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaPayloadType payload   = (*itr).first;
>-       std::string     expected  = (*itr).second;
>-
>-       std::string actual = mamaPayload_convertToString (payload);
>-
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       mamaPayloadType payload = itr->first;
>+       PayloadNameLib& namelib = itr->second;
>+       const char*     name    = namelib.first.c_str();
>+       const char*     library = namelib.second.c_str();
>+
>+       // Override for unknown, mama returns NULL for unknown library
>(so that
>+       // it prevents libraries from trying to load libmamaunknownmsg).
>+       if (MAMA_PAYLOAD_UNKNOWN == payload)
>+       {
>+           library = NULL;
>+       }
>+
>+       // Check name/payload conversion
>+       EXPECT_STREQ(name, mamaPayload_convertToString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromString
>(name));
>+
>+       // Check library/payload conversion
>+       EXPECT_STREQ(library, mamaPayload_convertToLibString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromLibString
>(library));
>     }
>-
>-    ASSERT_EQ( passed, true );
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertToString)
>+TEST_F (MamaEnumTestsC, testMiddlewareStringConversions)
> {
>+    MamaMiddlewareMapType testData;
>+    testData[MAMA_MIDDLEWARE_WMW]        = "wmw";
>+    testData[MAMA_MIDDLEWARE_LBM]        = "lbm";
>+    testData[MAMA_MIDDLEWARE_TIBRV]      = "tibrv";
>+    testData[MAMA_MIDDLEWARE_AVIS]       = "AVIS";
>+    testData[MAMA_MIDDLEWARE_TICK42BLP]  = "tick42blp";
>+    testData[MAMA_MIDDLEWARE_SOLACE]     = "SOLACE";
>+    testData[MAMA_MIDDLEWARE_RAI]        = "rai";
>+    testData[MAMA_MIDDLEWARE_QPID]       = "QPID";
>+    testData[MAMA_MIDDLEWARE_EXEGY]      = "exegy";
>+    testData[MAMA_MIDDLEWARE_IBMWFO]     = "ibmwfo";
>+    testData[MAMA_MIDDLEWARE_ACTIV]      = "activ";
>+    testData[MAMA_MIDDLEWARE_TICK42RMDS] = "tick42rmds";
>+    testData[MAMA_MIDDLEWARE_UMS]        = "ums";
>+    testData[MAMA_MIDDLEWARE_VULCAN]     = "vulcan";
>+    testData[MAMA_MIDDLEWARE_UNKNOWN]    = "unknown";
>     MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); ++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaMiddleware middleware = (*itr).first;
>-       std::string    expected   = (*itr).second;
>-
>-       std::string actual = mamaMiddleware_convertToString (middleware);
>+       mamaMiddleware middleware = itr->first;
>+       const char*    name       = itr->second.c_str();
>
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       EXPECT_STREQ(name, mamaMiddleware_convertToString (middleware));
>+       EXPECT_EQ((char)middleware,
>(char)mamaMiddleware_convertFromString (name));
>     }
>-
>-    ASSERT_EQ (passed, true);
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertFromString)
>-{
>-    MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>-
>-    for (itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); itr++) {
>-
>-        mamaMiddleware expected    = (*itr).first;
>-        std::string    middleware  = (*itr).second;
>-
>-        mamaMiddleware actual      = mamaMiddleware_convertFromString
>(middleware.c_str());
>-
>-        EXPECT_EQ (actual, expected);
>-
>-        if (actual != expected)
>-            passed = false;
>-    }
>-
>-    ASSERT_EQ (passed, true);
>-}
>

________________________________________________________

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.
________________________________________________________



 

--

Lee Skillen

 

Vulcan Financial Technologies

51 Malone Road, Belfast, BT9 6RY

 

Web:     www.vulcanft.com 



 

--

Lee Skillen

 

Vulcan Financial Technologies

51 Malone Road, Belfast, BT9 6RY

 

Office:  +44 (0)28 95 817888

Mobile:  +44 (0)78 41 425152

Web:     www.vulcanft.com 


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.



Glenn McClements <gmcclements@...>
 

I'm not too keen on the mapping file idea if they are user configurable, because that add to deployment headaches. I had thought of adding a query call to middleware/payload bridges which would allow them to self describe themselves, but this would require bridge changes.  A halfway house would be to link the text files into the .so/.dll as metadata which would allow us to query it at load time without bridge changes. This may also be a goo play to add, version info, display name info etc. 

 > I guess backwards compat is a concern for code out there that still refers to enumerations for some bridge-specific hacks or something else.
We will always (or for the foreseeable future anyway) need to keep the enums for backward compatibility, but for newer enums I would hope it could be dynamic. 

>A reasonable question is, without enumerations, what does the code in the bridges refer to? 
We could have explicit strings, which would map to a library name, as it is today, and then a handle (enum value effectively) would be returned to avoid string lookups in the future. 

>One approach could be, to keep it consistent with existing approaches maybe we can have a “PAYLOAD_PATH” 
….which could be generalised to OPENMAMA_LIB_PATH or similar, which would allow all OpenMAMA plugins to be loaded from one location. 

From: Sanjeev Wahi <wahi@...>
Organization: Realtime Data Services, LLC.
Date: Tuesday, 25 February 2014 14:02
To: Nyse Corp <gmcclements@...>, 'Lee Skillen' <lskillen@...>, Damian Maguire <DMaguire@...>
Cc: "openmama-dev@..." <openmama-dev@...>
Subject: RE: [Openmama-dev] [PATCH] MAMA: Add the capability to load payload libraries just-in-time (JIT).

One approach could be, to keep it consistent with existing approaches maybe we can have a “PAYLOAD_PATH” an environment variable pointing to a directories path containing payload info. *(Just like WOMBAT_PATH is being searched for mama.properties)

 

-Regards,

Sanjeev

 

 

Copyright © 2014 Realtime Data Services, LLC. All rights reserved. | www.RtdsLLC.com

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Glenn McClements

Sent: Tuesday, February 25, 2014 5:57 AM
To: Lee Skillen; Damian Maguire
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] [PATCH] MAMA: Add the capability to load payload libraries just-in-time (JIT).

 

Hi Lee,

At first glance the code looks good, and this is nice little feature with no adverse effects that I can think of. My only issue is that if we're tinkering with this area we should:

  • Go the whole hog and make payloads completely dynamic, removing the enum dependancy. Something like point OpenMAMA to a directory and it automatically loading all payloads in there, and query the payload on startup for it's identifier, not allowing two with the same identifier. There's some security issues with this I suppose.
  • Make it applicable to middleware (and any future kind of) bridges as well.

 

Happy to hear other peoples opinion on this. 

 

Cheers, 

Glenn 

 

From: Lee Skillen <lskillen@...>
Date: Thursday, 20 February 2014 13:42
To: Damian Maguire <DMaguire@...>
Cc: "openmama-dev@..." <openmama-dev@...>
Subject: Re: [Openmama-dev] [PATCH] MAMA: Add the capability to load payload libraries just-in-time (JIT).

 

Hey,

 

Just a quick update, it looks like our SMTP server caused the carriage returns - I've modified a version of git-send-email to include the capability to mime encode attachments as base64.  Tested and it carries through the patch in verbatim, so I shouldn't be sending any more patches that look like they came from a Windows machine. ;-)  I'll tidy up the git patch and submit it back to git-core next week since it seems useful (and may be useful to others here too).  Let me know if you need to me to re-submit the OpenMAMA patches I've already sent.

 

Thanks,
Lee

 

 

On 20 February 2014 17:08, Lee Skillen <lskillen@...> wrote:

Hey Damian,

 

Thanks, we're happy to finally be in a better position to start contributing.   First of all, just so that focus isn't purely on the JIT loading, I wanted to point out that this patch also has fixes for memory leaks, a reworking of the payload conversion unit tests, and a new style for the payload mappings (which we hope will be useful from a maintenance perspective to keep things centralised).

 

The new unit tests exercise the the string conversion capability and we ran the entire suite to ensure that nothing else has been broken.  Memory leaks are confirmed for presence before patch via valgrind and memory sanitizer and their removal is confirmed after.  We also utilise helgrind and thread sanitizer for threading issues in general, although not for this patch.

 

The JIT loading mechanism has been tested locally using a modified version of avis and our middleware.   Setup for a simple test would be to load a consuming application (e.g. mamaconsumerc) with any middleware that is payload agnostic, and then send a message utilising a payload that is different than the default payload for that middleware.  Pre-patch this would crash the application, but post-patch it'll load the payload just-in-time.

 

For testing the receiving bit is fine but none of the testtools applications have the capability to specify a different payload other than the default. So for the sake of demonstration I modified mamaproducerc to be capable of specifying payload on the command-line via -P <payload>.  The test looks like this:


Pre-patch:

 

$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v

Segmentation fault (core dumped)

 

Post-patch:

 

$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
[snip]
2014-02-20 16:57:20: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]

 

$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]
Starting to publish

 

Looking back at the mamaconsumer log we can see that it successfully loaded the vulcanmsg payload JIT :-

 

2014-02-20 16:58:09: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
Rate: 2 Latency(usec): Low: 13775       Average: 447183 High: 910969    Gaps: 1
Rate: 9 Latency(usec): Low: 34754       Average: 507466 High: 933766    Gaps: 0
Rate: 9 Latency(usec): Low: 56764       Average: 528090 High: 954780    Gaps: 0

 

Hope that looks good for you guys - Please feel free to experiment on the new branch.  We'd really like to get this code in to the mainline if at all possible, and we've got quite a lot more code to contribute too.

 

Apologies about the carriage returns in the patch file - I don't personally use Windows and we utilise git-format-patch and git-send-email for the patch formatting and sending.  The patch prior to sending doesn't have any carriage return characters in it, but I looked at one I pulled back and it does have 0x0d 0x0a ("\r\n") for the line endings.  So it seems that either SMTP or git-send-email was the culprit.  I'll have a look into preventing it but in the short-term `git apply --whitespace=fix` should sort it. :-(

 

Cracking start for submitting patches, eh. :-)

 

Cheers,

Lee

 

 

 

On 20 February 2014 14:36, Damian Maguire <DMaguire@...> wrote:

Cheers for this Lee, looks very interesting. From a quick look/preliminary
test the changes look solid, but given the size and scope of the change I
think the best thing to do is fire it into it's own branch for a while and
let people check it out for themselves. As such I'm going to stick it into
a feature-jit-payload branch on the main repo. Anyone who's interested can
then have a look and comment on the changes, and we can fix any bugs there
as required.

Regarding testing these changes, have you guys actually got this running
with any of the open source bridges, or do you have any sample testing you
can show us to demonstrate that it both works, and doesn't break the
existing code?

Just a quick note on the patch files you've attached - they seem to have
landed (to me at least) with DOS line endings, which messes with the git
apply step. It's not a big deal, since I can easily enough strip the line
endings, but probably something to watch out for with future submissions
(particularly if you touch any of the unfortunate files that already mix
and match line endings).

I'm also taking a look at the other patches you've sent through, so I'll
get back to you on those.

Cheers,

Damian




On 2/19/14 3:53 PM, "Lee Skillen" <lskillen@...> wrote:

>
>If a library hasn't been previously loaded then OpenMAMA will now attempt
>to load it by converting the payload identifier into a well-known string
>for the payload library.  This behaviour is useful if the payload isn't
>known ahead of time or if lazy loading is desired over enumeration of a
>existing set of libraries.  No changes made to existing API interfaces.
>
>The following functions have been added :-
>
>- mamaPayload_convertToLibString():
>  Maps a mamaPayloadType enumeration to the well-known library name.
>
>- mamaPayload_convertFromLibString():
>  Inverse of mamaPayload_convertToLibString.
>
>- mama_loadPayloadBridgeByType():
>  Takes a mamaPayloadType and utilises mamaPayload_convertToLibString()
>  to retrieve the library name for the payload, and then delegates this
>  to the existing mama_loadPayloadBridge() function.
>
>The string conversions are implemented as a linear search, primarily due
>to these functions not being on the hot path - Their use is prior to a
>library being loaded and (I would imagine) would be seldom used in other
>situations.
>
>Additional changes :-
>
>If msgPayload_destroyImpl has been exposed by the payload library then
>this is now utilised to cleanup the payload upon shutdown properly.  It
>was observed that each payload always leaked memory due to lacking the
>capability to destroy itself.  Individual payloads will still need to
>expose this - Backwards compatible as it won't be called otherwise.
>
>The MamaEnumTestsC set of unit tests have been rewritten to utilise the
>new conversion functionality.  The tests have also been renamed as such:
>
>- testPayloadToString -> testPayloadStringConversions
>- testPayloadFromString -> testPayloadStringConversions
>- testMiddlewareToString -> testMiddlewareStringConversions
>- testMiddlewareFromString -> testMiddlewareStringConversions
>
>Noticed that the ASSERT_EQ/EXPECT_EQ statements in gtest are sometimes
>used incorrectly, in that the expected/actual are reversed (it should be
>expected on the left and actual on the right).
>
>Todo :-
>
>Need to confirm well-known library names with all vendors - Will only be
>important if JIT loading is actually used, but the names should be made
>accurate anyway.
>
>Signed-off-by: Lee Skillen <lskillen@...>
>---
> mama/c_cpp/src/c/mama.c                            |  77 ++++++++--
> mama/c_cpp/src/c/mama/mama.h                       |  27 +++-
> mama/c_cpp/src/c/mama/msg.h                        |  30 ++++
> mama/c_cpp/src/c/msg.c                             | 163
>+++++++++++++++------
> mama/c_cpp/src/c/payloadbridge.h                   |   4 +
> .../src/gunittest/c/payloadmiddlewareidtest.cpp    | 152
>++++++++-----------
> 6 files changed, 304 insertions(+), 149 deletions(-)
>
>diff --git a/mama/c_cpp/src/c/mama.c b/mama/c_cpp/src/c/mama.c
>index 33a0768..3a7d7be 100644
>--- a/mama/c_cpp/src/c/mama.c
>+++ b/mama/c_cpp/src/c/mama.c
>@@ -28,6 +28,7 @@
> #include "wombat/strutils.h"
>
> #include <mama/mama.h>
>+#include <mama/msg.h>
> #include <mama/error.h>
> #include <mamainternal.h>
> #include <mama/version.h>
>@@ -168,6 +169,7 @@ typedef struct mamaImpl_
>     mamaPayloadBridge      myPayloads[MAMA_PAYLOAD_MAX];
>     LIB_HANDLE             myBridgeLibraries[MAMA_MIDDLEWARE_MAX];
>     LIB_HANDLE             myPayloadLibraries[MAMA_PAYLOAD_MAX];
>+    msgPayload_destroyImpl myPayloadDestroyFuncs[MAMA_PAYLOAD_MAX];
>     unsigned int           myRefCount;
>     wthread_static_mutex_t myLock;
> } mamaImpl;
>@@ -175,7 +177,7 @@ typedef struct mamaImpl_
> static mamaApplicationContext  appContext;
> static char mama_ver_string[256];
>
>-static mamaImpl gImpl = {{0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>+static mamaImpl gImpl = {{0}, {0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>
> /*
>*************************************************************************
>*/
> /* Private Function Prototypes. */
>@@ -624,10 +626,21 @@ mamaInternal_findBridge ()
> mamaPayloadBridge
> mamaInternal_findPayload (char id)
> {
>+    mamaPayloadBridge bridge = NULL;
>+
>     if ('\0' == id)
>         return NULL;
>
>-    return gImpl.myPayloads[(uint8_t)id];
>+    bridge = gImpl.myPayloads[(uint8_t)id];
>+
>+    if (NULL == bridge)
>+    {
>+        /* Dynamically load the payload bridge just-in-time, if it
>hasn't
>+         * already been loaded previously. */
>+        mama_loadPayloadBridgeByType (&bridge, (mamaPayloadType)id);
>+    }
>+
>+    return bridge;
> }
>
> mamaPayloadBridge
>@@ -1241,17 +1254,33 @@ mama_closeCount (unsigned int* count)
>
>         cleanupReservedFields();
>
>-         /* Look for a bridge for each of the payloads and close them */
>+        /* Look for a bridge for each of the payloads and close them */
>         for (payload = 0; payload != MAMA_PAYLOAD_MAX; ++payload)
>         {
>-              /* mamaPayloadBridgeImpl* impl = (mamaPayloadBridgeImpl*)
>-             * gImpl.myPayloads [(uint8_t)payload];*/
>-            gImpl.myPayloads[(uint8_t)payload] = NULL;
>-            if(gImpl.myPayloadLibraries[(uint8_t)payload])
>+            uint8_t                id   = (uint8_t)payload;
>+            LIB_HANDLE             lib  = gImpl.myPayloadLibraries[id];
>+            mamaPayloadBridgeImpl* impl = gImpl.myPayloads[id];
>+
>+            if (!lib)
>             {
>-                closeSharedLib
>(gImpl.myPayloadLibraries[(uint8_t)payload]);
>-                gImpl.myPayloadLibraries[(uint8_t)payload] = NULL;
>+                continue;
>+            }
>+
>+            if (impl)
>+            {
>+                msgPayload_destroyImpl destroyFunc =
>+                    gImpl.myPayloadDestroyFuncs[id];
>+
>+                if (destroyFunc)
>+                {
>+                    destroyFunc (impl);
>+                }
>+
>+                gImpl.myPayloads[id] = NULL;
>             }
>+
>+            closeSharedLib (lib);
>+            gImpl.myPayloadLibraries[id] = NULL;
>         }
>
>        gDefaultPayload = NULL;
>@@ -1827,18 +1856,29 @@ mama_setDefaultPayload (char id)
>
> mama_status
> mama_loadPayloadBridge (mamaPayloadBridge* impl,
>-                         const char*        payloadName)
>+                        const char*        payloadName)
> {
>     return mama_loadPayloadBridgeInternal (impl, payloadName);
> }
>+
>+mama_status
>+mama_loadPayloadBridgeByType (mamaPayloadBridge* impl,
>+                              mamaPayloadType    payload)
>+{
>+    const char* name = mamaPayload_convertToLibString (payload);
>+    return mama_loadPayloadBridgeInternal (impl, name);
>+}
>+
> mama_status
> mama_loadPayloadBridgeInternal  (mamaPayloadBridge* impl,
>                                  const char*        payloadName)
> {
>     char                    bridgeImplName  [256];
>     char                    initFuncName    [256];
>+    char                    destroyFuncName [256];
>     LIB_HANDLE              bridgeLib       = NULL;
>     msgPayload_createImpl   initFunc        = NULL;
>+    msgPayload_destroyImpl  destroyFunc     = NULL;
>     mama_status             status          = MAMA_STATUS_OK;
>     char                    payloadChar       ='\0';
>       void*                                   vp                              = NULL;
>@@ -1865,11 +1905,10 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>         return MAMA_STATUS_NO_BRIDGE_IMPL;
>     }
>
>-    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>-
>     /* Gives a warning - casting from void* to bridge_createImpl func */
>-      vp = loadLibFunc (bridgeLib, initFuncName);
>-      initFunc  = *(msgPayload_createImpl*) &vp;
>+    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>+    vp = loadLibFunc (bridgeLib, initFuncName);
>+    initFunc  = *(msgPayload_createImpl*) &vp;
>
>     if (!initFunc)
>     {
>@@ -1887,7 +1926,7 @@ mama_loadPayloadBridgeInternal  (mamaPayloadBridge*
>impl,
>
>     if (MAMA_STATUS_OK != (status = initFunc (impl, &payloadChar)))
>     {
>-       wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return status;
>     }
>@@ -1910,13 +1949,19 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>              "Payload bridge %s already loaded",
>              payloadName);
>
>-         wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return MAMA_STATUS_OK;
>     }
>
>+    /* Gives a warning - casting from void* to bridge_destroyImpl func */
>+    snprintf (destroyFuncName, 256, "%sPayload_destroyImpl",
>payloadName);
>+    vp = loadLibFunc (bridgeLib, destroyFuncName);
>+    destroyFunc = *(msgPayload_destroyImpl*) &vp;
>+
>     gImpl.myPayloads [(int)payloadChar] = *impl;
>     gImpl.myPayloadLibraries [(int)payloadChar] = bridgeLib;
>+    gImpl.myPayloadDestroyFuncs [(int)payloadChar] = destroyFunc;
>
>     if (!gDefaultPayload)
>     {
>diff --git a/mama/c_cpp/src/c/mama/mama.h b/mama/c_cpp/src/c/mama/mama.h
>index a574f36..07a9115 100644
>--- a/mama/c_cpp/src/c/mama/mama.h
>+++ b/mama/c_cpp/src/c/mama/mama.h
>@@ -133,11 +133,34 @@ extern "C"
>      */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadBridge (mamaBridge* impl, const char*  middleware);
>+    mama_loadBridge (mamaBridge* impl,
>+                     const char* middleware);
>
>+    /**
>+     * Load a payload bridge by its well-known library name string.
>E.g. "wmsg"
>+     * will load the WombatMsg payload, provided it exists on the system.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payloadName The well-known name of the payload library.
>+     */
>+    MAMAExpDLL
>+    extern mama_status
>+    mama_loadPayloadBridge (mamaPayloadBridge* bridge,
>+                            const char*        payloadName);
>+
>+    /**
>+     * Load a payload bridge by using the mamaPayloadType identifier.
>This
>+     * utilises mamaPayload_convertToLibString() in order to retrieve the
>+     * well-known library string for the payload library, and this is
>then
>+     * delegated to mama_loadPayloadBridge.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payload The payload enumeration identifier.
>+     */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadPayloadBridge (mamaPayloadBridge*  bridge,  const char*
>payloadName);
>+    mama_loadPayloadBridgeByType (mamaPayloadBridge* bridge,
>+                                  mamaPayloadType    payload);
>
>     /** Load the bridge specified by middleware string using the path
>specified by the user.
>      * If the bridge has already been loaded then the existing bridge
>instance
>diff --git a/mama/c_cpp/src/c/mama/msg.h b/mama/c_cpp/src/c/mama/msg.h
>index bf0d562..a9c3438 100644
>--- a/mama/c_cpp/src/c/mama/msg.h
>+++ b/mama/c_cpp/src/c/mama/msg.h
>@@ -74,6 +74,36 @@ extern const char*
> mamaPayload_convertToString (mamaPayloadType payloadType);
>
> /**
>+ * Convert a string to a mamaPayloadType.
>+ *
>+ * @param str The str to convert.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromString (const char* str);
>+
>+/**
>+ * Convert a mamaPayloadType value to a well-known library name string
>+ * for that payload.  E.g. MAMA_PAYLOAD_WOMBAT_MSG translates into
>+ * "wmsg" (of which the payload library would be libwmsgimpl.so on *nix).
>+ *
>+ * @param payloadType The payloadType to convert to a library name
>string.
>+ */
>+MAMAExpDLL
>+extern const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType);
>+
>+/**
>+ * Convert a well-known library name to a mamaPayloadType value
>+ * for that payload.  E.g. "wmsg" translates into
>MAMA_PAYLOAD_WOMBAT_MSG.
>+ *
>+ * @param str The library name string to convert to a mamaPayloadType.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str);
>+
>+/**
>  * Create a mamaMsg.
>  *
>  * @param msg The location of a mamaMsg where to store the result.
>diff --git a/mama/c_cpp/src/c/msg.c b/mama/c_cpp/src/c/msg.c
>index 286fcad..506c390 100644
>--- a/mama/c_cpp/src/c/msg.c
>+++ b/mama/c_cpp/src/c/msg.c
>@@ -41,13 +41,39 @@
>
> #include "wombat/wincompat.h"
>
>-
> #define MAX_SUBJECT 256
>
>-#define NOFID 0
>-
>-static const int INCLUDE_FIELD_NAME = (1 << 4);
>-static const int INCLUDE_FIELD_ID   = (1 << 2);
>+typedef struct PayloadTable
>+{
>+    mamaPayloadType mType;
>+    const char*     mName;
>+    const char*     mLibrary;
>+}
>+PayloadTable;
>+
>+static
>+const PayloadTable kPayloads [] =
>+{
>+    {MAMA_PAYLOAD_ACTIV,      "activ",      "activmsg"},
>+    {MAMA_PAYLOAD_AVIS,       "AVIS",       "avismsg"},
>+    {MAMA_PAYLOAD_EXEGY,      "EXEGY",      "exegymsg"},
>+    {MAMA_PAYLOAD_FAST,       "FAST",       "fastmsg"},
>+    {MAMA_PAYLOAD_IBMWFO,     "ibmwfo",     "ibmwfomsg"},
>+    {MAMA_PAYLOAD_INRUSH,     "INRUSH",     "inrushmsg"},
>+    {MAMA_PAYLOAD_QPID,       "QPID",       "qpidmsg"},
>+    {MAMA_PAYLOAD_RAI,        "rai",        "raimsg"},
>+    {MAMA_PAYLOAD_SOLACE,     "solacemsg",  "solacemsg"},
>+    {MAMA_PAYLOAD_TIBRV,      "TIBRV",      "tibrvmsg"},
>+    {MAMA_PAYLOAD_TICK42BLP,  "TICK42BLP",  "tick42blpmsg"},
>+    {MAMA_PAYLOAD_TICK42RMDS, "TICK42RMDS", "tick42rmdsmsg"},
>+    {MAMA_PAYLOAD_UMS,        "UMS",        "umsmsg"},
>+    {MAMA_PAYLOAD_V5,         "V5",         "wcachemsg"},
>+    {MAMA_PAYLOAD_VULCAN,     "Vulcan",     "vulcanmsg"},
>+    {MAMA_PAYLOAD_WOMBAT_MSG, "WombatMsg",  "wmsg"},
>+
>+    /* Linear search requires unknown to be at the end-of-table */
>+    {MAMA_PAYLOAD_UNKNOWN,    "unknown",    NULL}
>+};
>
> typedef struct mamaMsgIteratorImpl
> {
>@@ -106,6 +132,66 @@ typedef struct mamaMsgImpl_
> static mama_status
> mamaMsgImpl_destroyLastVectorMsg (mamaMsgImpl *impl);
>
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByType (mamaPayloadType payloadType)
>+{
>+    int k = 0;
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (kPayloads[k].mType == payloadType)
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByName (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mName))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByLibrary (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mLibrary))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
> /*=================================================================
>   = Public functions - defined in mama/msg.h
>   ================================================================*/
>@@ -166,7 +252,9 @@ mamaMsg_destroy (mamaMsg msg)
>     if (impl->mCurrentField)
>     {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>     }
>+
>     /*Destroy the reusable field object*/
>     if (impl->mCopy)
>     {
>@@ -366,44 +454,29 @@ mamaMsgImpl_getPayload (const mamaMsg msg,
>msgPayload* payload)
> const char*
> mamaPayload_convertToString (mamaPayloadType payloadType)
> {
>-    switch (payloadType)
>-    {
>-        case MAMA_PAYLOAD_SOLACE:
>-            return "solacemsg";
>-        case MAMA_PAYLOAD_V5:
>-            return "V5";
>-        case MAMA_PAYLOAD_AVIS:
>-            return "AVIS";
>-        case MAMA_PAYLOAD_TICK42BLP:
>-            return "TICK42BLP";
>-        case MAMA_PAYLOAD_FAST:
>-            return "FAST";
>-        case MAMA_PAYLOAD_RAI:
>-            return "rai";
>-        case MAMA_PAYLOAD_UMS:
>-            return "UMS";
>-        case MAMA_PAYLOAD_TICK42RMDS:
>-            return "TICK42RMDS";
>-        case MAMA_PAYLOAD_QPID:
>-            return "QPID";
>-        case MAMA_PAYLOAD_TIBRV:
>-            return "TIBRV";
>-        case MAMA_PAYLOAD_IBMWFO:
>-            return "ibmwfo";
>-        case MAMA_PAYLOAD_ACTIV:
>-            return "activ";
>-        case MAMA_PAYLOAD_VULCAN:
>-            return "Vulcan";
>-        case MAMA_PAYLOAD_WOMBAT_MSG:
>-            return "WombatMsg";
>-        case MAMA_PAYLOAD_EXEGY:
>-            return "EXEGY";
>-        case MAMA_PAYLOAD_INRUSH:
>-            return "INRUSH";
>-
>-        default:
>-            return "unknown";
>-    }
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mName;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByName (str);
>+    return table->mType;
>+}
>+
>+const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mLibrary;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByLibrary
>(str);
>+    return table->mType;
> }
>
> mama_status
>@@ -3481,7 +3554,11 @@ mama_status mamaMsgIterator_destroy
>(mamaMsgIterator iterator)
>     }
>
>     if (impl->mCurrentField)
>+    {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>+    }
>+
>     free (impl);
>     return MAMA_STATUS_OK;
> }
>diff --git a/mama/c_cpp/src/c/payloadbridge.h
>b/mama/c_cpp/src/c/payloadbridge.h
>index 703c1a9..6ad7108 100644
>--- a/mama/c_cpp/src/c/payloadbridge.h
>+++ b/mama/c_cpp/src/c/payloadbridge.h
>@@ -422,8 +422,12 @@ while(0)
>                       \
> typedef mama_status
> (*msgPayload_createImpl)      (mamaPayloadBridge* result, char*
>identifier);
>
>+typedef mama_status
>+(*msgPayload_destroyImpl)     (mamaPayloadBridge result);
>+
> typedef mamaPayloadType
> (*msgPayload_getType)         (void);
>+
> /*===================================================================
>  =              msgPayload bridge function pointers                 =
>  ====================================================================*/
>diff --git a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>index ed18a64..1e14227 100644
>--- a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>+++ b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>@@ -20,14 +20,9 @@ protected:
>
>    virtual void TearDown(void);
>
>-   typedef std::pair <mamaPayloadType, std::string> MamaPayloadPair;
>-   typedef std::map  <mamaPayloadType, std::string> MamaPayloadMapType;
>-
>-   typedef std::pair <mamaMiddleware,  std::string> MamaMiddlewarePair;
>-   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>-
>-   MamaPayloadMapType     payloadTestData;
>-   MamaMiddlewareMapType  middlewareTestData;
>+   typedef std::pair <std::string,     std::string>    PayloadNameLib;
>+   typedef std::map  <mamaPayloadType, PayloadNameLib>
>MamaPayloadMapType;
>+   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>
> private:
>    void CreateTestData();
>@@ -55,100 +50,81 @@ MamaEnumTestsC::TearDown()
> void
> MamaEnumTestsC::CreateTestData()
> {
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_V5,
>"V5"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_AVIS,
>"AVIS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42BLP,
>"TICK42BLP"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_FAST,
>"FAST"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_RAI,
>"rai"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UMS,
>"UMS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42RMDS,
>"TICK42RMDS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_QPID,
>"QPID"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TIBRV,
>"TIBRV"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_IBMWFO,
>"ibmwfo"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_ACTIV,
>"activ"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_VULCAN,
>"Vulcan"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_WOMBAT_MSG,
>"WombatMsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_EXEGY,
>"EXEGY"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_SOLACE,
>"solacemsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UNKNOWN,
>"unknown"));
>-
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_WMW,
>     "wmw"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_LBM,
>     "lbm"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TIBRV,     "tibrv"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_AVIS,
>     "AVIS"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42BLP, "tick42blp"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_SOLACE,    "SOLACE"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_RAI,
>     "rai"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_QPID,
>     "QPID"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_EXEGY,     "exegy"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_IBMWFO,    "ibmwfo"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_ACTIV,     "activ"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42RMDS, "tick42rmds"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_UMS,
>     "ums"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_VULCAN,    "vulcan"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_UNKNOWN,   "unknown"));
> }
>
>-
>-TEST_F (MamaEnumTestsC, testPayloadConvertToString)
>+TEST_F (MamaEnumTestsC, testPayloadStringConversions)
> {
>+    MamaPayloadMapType testData;
>+    testData[MAMA_PAYLOAD_ACTIV]      = PayloadNameLib ("activ",
>"activmsg");
>+    testData[MAMA_PAYLOAD_AVIS]       = PayloadNameLib ("AVIS",
>"avismsg");
>+    testData[MAMA_PAYLOAD_EXEGY]      = PayloadNameLib ("EXEGY",
>"exegymsg");
>+    testData[MAMA_PAYLOAD_FAST]       = PayloadNameLib ("FAST",
>"fastmsg");
>+    testData[MAMA_PAYLOAD_IBMWFO]     = PayloadNameLib ("ibmwfo",
>"ibmwfomsg");
>+    testData[MAMA_PAYLOAD_QPID]       = PayloadNameLib ("QPID",
>"qpidmsg");
>+    testData[MAMA_PAYLOAD_RAI]        = PayloadNameLib ("rai",
>"raimsg");
>+    testData[MAMA_PAYLOAD_SOLACE]     = PayloadNameLib ("solacemsg",
>"solacemsg");
>+    testData[MAMA_PAYLOAD_TIBRV]      = PayloadNameLib ("TIBRV",
>"tibrvmsg");
>+    testData[MAMA_PAYLOAD_TICK42BLP]  = PayloadNameLib ("TICK42BLP",
>"tick42blpmsg");
>+    testData[MAMA_PAYLOAD_TICK42RMDS] = PayloadNameLib ("TICK42RMDS",
>"tick42rmdsmsg");
>+    testData[MAMA_PAYLOAD_UMS]        = PayloadNameLib ("UMS",
>"umsmsg");
>+    testData[MAMA_PAYLOAD_VULCAN]     = PayloadNameLib ("Vulcan",
>"vulcanmsg");
>+    testData[MAMA_PAYLOAD_WOMBAT_MSG] = PayloadNameLib ("WombatMsg",
>"wmsg");
>+    testData[MAMA_PAYLOAD_V5]         = PayloadNameLib ("V5",
>"wcachemsg");
>+    testData[MAMA_PAYLOAD_UNKNOWN]    = PayloadNameLib ("unknown",
>"");
>+
>     MamaPayloadMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = payloadTestData.begin(); itr != payloadTestData.end();
>++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaPayloadType payload   = (*itr).first;
>-       std::string     expected  = (*itr).second;
>-
>-       std::string actual = mamaPayload_convertToString (payload);
>-
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       mamaPayloadType payload = itr->first;
>+       PayloadNameLib& namelib = itr->second;
>+       const char*     name    = namelib.first.c_str();
>+       const char*     library = namelib.second.c_str();
>+
>+       // Override for unknown, mama returns NULL for unknown library
>(so that
>+       // it prevents libraries from trying to load libmamaunknownmsg).
>+       if (MAMA_PAYLOAD_UNKNOWN == payload)
>+       {
>+           library = NULL;
>+       }
>+
>+       // Check name/payload conversion
>+       EXPECT_STREQ(name, mamaPayload_convertToString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromString
>(name));
>+
>+       // Check library/payload conversion
>+       EXPECT_STREQ(library, mamaPayload_convertToLibString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromLibString
>(library));
>     }
>-
>-    ASSERT_EQ( passed, true );
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertToString)
>+TEST_F (MamaEnumTestsC, testMiddlewareStringConversions)
> {
>+    MamaMiddlewareMapType testData;
>+    testData[MAMA_MIDDLEWARE_WMW]        = "wmw";
>+    testData[MAMA_MIDDLEWARE_LBM]        = "lbm";
>+    testData[MAMA_MIDDLEWARE_TIBRV]      = "tibrv";
>+    testData[MAMA_MIDDLEWARE_AVIS]       = "AVIS";
>+    testData[MAMA_MIDDLEWARE_TICK42BLP]  = "tick42blp";
>+    testData[MAMA_MIDDLEWARE_SOLACE]     = "SOLACE";
>+    testData[MAMA_MIDDLEWARE_RAI]        = "rai";
>+    testData[MAMA_MIDDLEWARE_QPID]       = "QPID";
>+    testData[MAMA_MIDDLEWARE_EXEGY]      = "exegy";
>+    testData[MAMA_MIDDLEWARE_IBMWFO]     = "ibmwfo";
>+    testData[MAMA_MIDDLEWARE_ACTIV]      = "activ";
>+    testData[MAMA_MIDDLEWARE_TICK42RMDS] = "tick42rmds";
>+    testData[MAMA_MIDDLEWARE_UMS]        = "ums";
>+    testData[MAMA_MIDDLEWARE_VULCAN]     = "vulcan";
>+    testData[MAMA_MIDDLEWARE_UNKNOWN]    = "unknown";
>     MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); ++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaMiddleware middleware = (*itr).first;
>-       std::string    expected   = (*itr).second;
>-
>-       std::string actual = mamaMiddleware_convertToString (middleware);
>+       mamaMiddleware middleware = itr->first;
>+       const char*    name       = itr->second.c_str();
>
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       EXPECT_STREQ(name, mamaMiddleware_convertToString (middleware));
>+       EXPECT_EQ((char)middleware,
>(char)mamaMiddleware_convertFromString (name));
>     }
>-
>-    ASSERT_EQ (passed, true);
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertFromString)
>-{
>-    MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>-
>-    for (itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); itr++) {
>-
>-        mamaMiddleware expected    = (*itr).first;
>-        std::string    middleware  = (*itr).second;
>-
>-        mamaMiddleware actual      = mamaMiddleware_convertFromString
>(middleware.c_str());
>-
>-        EXPECT_EQ (actual, expected);
>-
>-        if (actual != expected)
>-            passed = false;
>-    }
>-
>-    ASSERT_EQ (passed, true);
>-}
>

________________________________________________________

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.
________________________________________________________



 

--

Lee Skillen

 

Vulcan Financial Technologies

51 Malone Road, Belfast, BT9 6RY

 

Web:     www.vulcanft.com 



 

--

Lee Skillen

 

Vulcan Financial Technologies

51 Malone Road, Belfast, BT9 6RY

 

Office:  +44 (0)28 95 817888

Mobile:  +44 (0)78 41 425152

Web:     www.vulcanft.com 


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.



This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


Sanjeev Wahi <wahi@...>
 

Also, may be at this stage of Re-Architecting we can further decouple default OpenMAMA API’s dependence on WOMBAT references /nomenclature ?

 

Following is the current OpenMAMA API default from mama-developers-guige-c++.pdf Page#19:

 

4. Properties

There are a number of mechanisms in place by which properties for the MAMA API, and the underlying

messaging middleware specific APIs, can be specified.

The default behavior for the API (for all languages) is to look for a file named mama.properties in the

directory specified by the WOMBAT_PATH environment variable. If present, the specified properties file is

loaded on application startup when Mama::open() is called. MAMA accesses the properties file and

looks for all the possible properties (see Section 18: Configuration Reference for details).

 

 

 

If we are considering Re-Architecting how default API (Bridge / Middleware / PayLoad) gets loaded. May be then have another generic name variable something similar to “PAYLOAD_PATH” or “OPENMAMA_LIB_PATH” as  default (although still continue supporting WOMBAT_PATH for existing users).

 

This will further decouple the default dependence of OpenMAMA APIs on WOMBAT nomenclature/references.

 

Plus this will give a place to redefine how the default dynamic loading of Bridge/Middleware/Payload should be done. Now here we can decide how much dynamic information gests compiled in .so/.dll & also still some optimal user text/edit configurable scope, which comes very handy at times.

 

 

 

-Best Regards,

Sanjeev

 

 

 

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Tuesday, February 25, 2014 12:00 PM
To: Sanjeev Wahi; 'Lee Skillen'; Damian Maguire
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] [PATCH] MAMA: Add the capability to load payload libraries just-in-time (JIT).

 

I'm not too keen on the mapping file idea if they are user configurable, because that add to deployment headaches. I had thought of adding a query call to middleware/payload bridges which would allow them to self describe themselves, but this would require bridge changes.  A halfway house would be to link the text files into the .so/.dll as metadata which would allow us to query it at load time without bridge changes. This may also be a goo play to add, version info, display name info etc. 

 

 > I guess backwards compat is a concern for code out there that still refers to enumerations for some bridge-specific hacks or something else.

We will always (or for the foreseeable future anyway) need to keep the enums for backward compatibility, but for newer enums I would hope it could be dynamic. 

 

>A reasonable question is, without enumerations, what does the code in the bridges refer to? 

We could have explicit strings, which would map to a library name, as it is today, and then a handle (enum value effectively) would be returned to avoid string lookups in the future. 

 

>One approach could be, to keep it consistent with existing approaches maybe we can have a “PAYLOAD_PATH” 

….which could be generalised to OPENMAMA_LIB_PATH or similar, which would allow all OpenMAMA plugins to be loaded from one location. 

 

From: Sanjeev Wahi <wahi@...>
Organization: Realtime Data Services, LLC.
Date: Tuesday, 25 February 2014 14:02
To: Nyse Corp <gmcclements@...>, 'Lee Skillen' <lskillen@...>, Damian Maguire <DMaguire@...>
Cc: "openmama-dev@..." <openmama-dev@...>
Subject: RE: [Openmama-dev] [PATCH] MAMA: Add the capability to load payload libraries just-in-time (JIT).

 

One approach could be, to keep it consistent with existing approaches maybe we can have a “PAYLOAD_PATH” an environment variable pointing to a directories path containing payload info. *(Just like WOMBAT_PATH is being searched for mama.properties)

 

-Regards,

Sanjeev

 

 

Copyright © 2014 Realtime Data Services, LLC. All rights reserved. | www.RtdsLLC.com

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Glenn McClements
Sent: Tuesday, February 25, 2014 5:57 AM
To: Lee Skillen; Damian Maguire
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] [PATCH] MAMA: Add the capability to load payload libraries just-in-time (JIT).

 

Hi Lee,

At first glance the code looks good, and this is nice little feature with no adverse effects that I can think of. My only issue is that if we're tinkering with this area we should:

  • Go the whole hog and make payloads completely dynamic, removing the enum dependancy. Something like point OpenMAMA to a directory and it automatically loading all payloads in there, and query the payload on startup for it's identifier, not allowing two with the same identifier. There's some security issues with this I suppose.
  • Make it applicable to middleware (and any future kind of) bridges as well.

 

Happy to hear other peoples opinion on this. 

 

Cheers, 

Glenn 

 

From: Lee Skillen <lskillen@...>
Date: Thursday, 20 February 2014 13:42
To: Damian Maguire <DMaguire@...>
Cc: "openmama-dev@..." <openmama-dev@...>
Subject: Re: [Openmama-dev] [PATCH] MAMA: Add the capability to load payload libraries just-in-time (JIT).

 

Hey,

 

Just a quick update, it looks like our SMTP server caused the carriage returns - I've modified a version of git-send-email to include the capability to mime encode attachments as base64.  Tested and it carries through the patch in verbatim, so I shouldn't be sending any more patches that look like they came from a Windows machine. ;-)  I'll tidy up the git patch and submit it back to git-core next week since it seems useful (and may be useful to others here too).  Let me know if you need to me to re-submit the OpenMAMA patches I've already sent.

 

Thanks,
Lee

 

 

On 20 February 2014 17:08, Lee Skillen <lskillen@...> wrote:

Hey Damian,

 

Thanks, we're happy to finally be in a better position to start contributing.   First of all, just so that focus isn't purely on the JIT loading, I wanted to point out that this patch also has fixes for memory leaks, a reworking of the payload conversion unit tests, and a new style for the payload mappings (which we hope will be useful from a maintenance perspective to keep things centralised).

 

The new unit tests exercise the the string conversion capability and we ran the entire suite to ensure that nothing else has been broken.  Memory leaks are confirmed for presence before patch via valgrind and memory sanitizer and their removal is confirmed after.  We also utilise helgrind and thread sanitizer for threading issues in general, although not for this patch.

 

The JIT loading mechanism has been tested locally using a modified version of avis and our middleware.   Setup for a simple test would be to load a consuming application (e.g. mamaconsumerc) with any middleware that is payload agnostic, and then send a message utilising a payload that is different than the default payload for that middleware.  Pre-patch this would crash the application, but post-patch it'll load the payload just-in-time.

 

For testing the receiving bit is fine but none of the testtools applications have the capability to specify a different payload other than the default. So for the sake of demonstration I modified mamaproducerc to be capable of specifying payload on the command-line via -P <payload>.  The test looks like this:


Pre-patch:

 

$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v

Segmentation fault (core dumped)

 

Post-patch:

 

$ ./mamaconsumerc -m avis -s MAMA_TOPIC -i 10 -v
[snip]
2014-02-20 16:57:20: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]

 

$ ./mamaproducerc -m avis -P vulcanmsg -sleep 100000000 -v
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
[snip]
2014-02-20 16:57:44: mama_loadPayloadBridge(): Sucessfully loaded avismsg payload bridge from library [mamaavismsgimpl]
[snip]
Starting to publish

 

Looking back at the mamaconsumer log we can see that it successfully loaded the vulcanmsg payload JIT :-

 

2014-02-20 16:58:09: mama_loadPayloadBridge(): Sucessfully loaded vulcanmsg payload bridge from library [mamavulcanmsgimpl]
Rate: 2 Latency(usec): Low: 13775       Average: 447183 High: 910969    Gaps: 1
Rate: 9 Latency(usec): Low: 34754       Average: 507466 High: 933766    Gaps: 0
Rate: 9 Latency(usec): Low: 56764       Average: 528090 High: 954780    Gaps: 0

 

Hope that looks good for you guys - Please feel free to experiment on the new branch.  We'd really like to get this code in to the mainline if at all possible, and we've got quite a lot more code to contribute too.

 

Apologies about the carriage returns in the patch file - I don't personally use Windows and we utilise git-format-patch and git-send-email for the patch formatting and sending.  The patch prior to sending doesn't have any carriage return characters in it, but I looked at one I pulled back and it does have 0x0d 0x0a ("\r\n") for the line endings.  So it seems that either SMTP or git-send-email was the culprit.  I'll have a look into preventing it but in the short-term `git apply --whitespace=fix` should sort it. :-(

 

Cracking start for submitting patches, eh. :-)

 

Cheers,

Lee

 

 

 

On 20 February 2014 14:36, Damian Maguire <DMaguire@...> wrote:

Cheers for this Lee, looks very interesting. From a quick look/preliminary
test the changes look solid, but given the size and scope of the change I
think the best thing to do is fire it into it's own branch for a while and
let people check it out for themselves. As such I'm going to stick it into
a feature-jit-payload branch on the main repo. Anyone who's interested can
then have a look and comment on the changes, and we can fix any bugs there
as required.

Regarding testing these changes, have you guys actually got this running
with any of the open source bridges, or do you have any sample testing you
can show us to demonstrate that it both works, and doesn't break the
existing code?

Just a quick note on the patch files you've attached - they seem to have
landed (to me at least) with DOS line endings, which messes with the git
apply step. It's not a big deal, since I can easily enough strip the line
endings, but probably something to watch out for with future submissions
(particularly if you touch any of the unfortunate files that already mix
and match line endings).

I'm also taking a look at the other patches you've sent through, so I'll
get back to you on those.

Cheers,

Damian




On 2/19/14 3:53 PM, "Lee Skillen" <lskillen@...> wrote:

>
>If a library hasn't been previously loaded then OpenMAMA will now attempt
>to load it by converting the payload identifier into a well-known string
>for the payload library.  This behaviour is useful if the payload isn't
>known ahead of time or if lazy loading is desired over enumeration of a
>existing set of libraries.  No changes made to existing API interfaces.
>
>The following functions have been added :-
>
>- mamaPayload_convertToLibString():
>  Maps a mamaPayloadType enumeration to the well-known library name.
>
>- mamaPayload_convertFromLibString():
>  Inverse of mamaPayload_convertToLibString.
>
>- mama_loadPayloadBridgeByType():
>  Takes a mamaPayloadType and utilises mamaPayload_convertToLibString()
>  to retrieve the library name for the payload, and then delegates this
>  to the existing mama_loadPayloadBridge() function.
>
>The string conversions are implemented as a linear search, primarily due
>to these functions not being on the hot path - Their use is prior to a
>library being loaded and (I would imagine) would be seldom used in other
>situations.
>
>Additional changes :-
>
>If msgPayload_destroyImpl has been exposed by the payload library then
>this is now utilised to cleanup the payload upon shutdown properly.  It
>was observed that each payload always leaked memory due to lacking the
>capability to destroy itself.  Individual payloads will still need to
>expose this - Backwards compatible as it won't be called otherwise.
>
>The MamaEnumTestsC set of unit tests have been rewritten to utilise the
>new conversion functionality.  The tests have also been renamed as such:
>
>- testPayloadToString -> testPayloadStringConversions
>- testPayloadFromString -> testPayloadStringConversions
>- testMiddlewareToString -> testMiddlewareStringConversions
>- testMiddlewareFromString -> testMiddlewareStringConversions
>
>Noticed that the ASSERT_EQ/EXPECT_EQ statements in gtest are sometimes
>used incorrectly, in that the expected/actual are reversed (it should be
>expected on the left and actual on the right).
>
>Todo :-
>
>Need to confirm well-known library names with all vendors - Will only be
>important if JIT loading is actually used, but the names should be made
>accurate anyway.
>
>Signed-off-by: Lee Skillen <lskillen@...>
>---
> mama/c_cpp/src/c/mama.c                            |  77 ++++++++--
> mama/c_cpp/src/c/mama/mama.h                       |  27 +++-
> mama/c_cpp/src/c/mama/msg.h                        |  30 ++++
> mama/c_cpp/src/c/msg.c                             | 163
>+++++++++++++++------
> mama/c_cpp/src/c/payloadbridge.h                   |   4 +
> .../src/gunittest/c/payloadmiddlewareidtest.cpp    | 152
>++++++++-----------
> 6 files changed, 304 insertions(+), 149 deletions(-)
>
>diff --git a/mama/c_cpp/src/c/mama.c b/mama/c_cpp/src/c/mama.c
>index 33a0768..3a7d7be 100644
>--- a/mama/c_cpp/src/c/mama.c
>+++ b/mama/c_cpp/src/c/mama.c
>@@ -28,6 +28,7 @@
> #include "wombat/strutils.h"
>
> #include <mama/mama.h>
>+#include <mama/msg.h>
> #include <mama/error.h>
> #include <mamainternal.h>
> #include <mama/version.h>
>@@ -168,6 +169,7 @@ typedef struct mamaImpl_
>     mamaPayloadBridge      myPayloads[MAMA_PAYLOAD_MAX];
>     LIB_HANDLE             myBridgeLibraries[MAMA_MIDDLEWARE_MAX];
>     LIB_HANDLE             myPayloadLibraries[MAMA_PAYLOAD_MAX];
>+    msgPayload_destroyImpl myPayloadDestroyFuncs[MAMA_PAYLOAD_MAX];
>     unsigned int           myRefCount;
>     wthread_static_mutex_t myLock;
> } mamaImpl;
>@@ -175,7 +177,7 @@ typedef struct mamaImpl_
> static mamaApplicationContext  appContext;
> static char mama_ver_string[256];
>
>-static mamaImpl gImpl = {{0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>+static mamaImpl gImpl = {{0}, {0}, {0}, {0}, {0}, 0,
>WSTATIC_MUTEX_INITIALIZER};
>
> /*
>*************************************************************************
>*/
> /* Private Function Prototypes. */
>@@ -624,10 +626,21 @@ mamaInternal_findBridge ()
> mamaPayloadBridge
> mamaInternal_findPayload (char id)
> {
>+    mamaPayloadBridge bridge = NULL;
>+
>     if ('\0' == id)
>         return NULL;
>
>-    return gImpl.myPayloads[(uint8_t)id];
>+    bridge = gImpl.myPayloads[(uint8_t)id];
>+
>+    if (NULL == bridge)
>+    {
>+        /* Dynamically load the payload bridge just-in-time, if it
>hasn't
>+         * already been loaded previously. */
>+        mama_loadPayloadBridgeByType (&bridge, (mamaPayloadType)id);
>+    }
>+
>+    return bridge;
> }
>
> mamaPayloadBridge
>@@ -1241,17 +1254,33 @@ mama_closeCount (unsigned int* count)
>
>         cleanupReservedFields();
>
>-         /* Look for a bridge for each of the payloads and close them */
>+        /* Look for a bridge for each of the payloads and close them */
>         for (payload = 0; payload != MAMA_PAYLOAD_MAX; ++payload)
>         {
>-              /* mamaPayloadBridgeImpl* impl = (mamaPayloadBridgeImpl*)
>-             * gImpl.myPayloads [(uint8_t)payload];*/
>-            gImpl.myPayloads[(uint8_t)payload] = NULL;
>-            if(gImpl.myPayloadLibraries[(uint8_t)payload])
>+            uint8_t                id   = (uint8_t)payload;
>+            LIB_HANDLE             lib  = gImpl.myPayloadLibraries[id];
>+            mamaPayloadBridgeImpl* impl = gImpl.myPayloads[id];
>+
>+            if (!lib)
>             {
>-                closeSharedLib
>(gImpl.myPayloadLibraries[(uint8_t)payload]);
>-                gImpl.myPayloadLibraries[(uint8_t)payload] = NULL;
>+                continue;
>+            }
>+
>+            if (impl)
>+            {
>+                msgPayload_destroyImpl destroyFunc =
>+                    gImpl.myPayloadDestroyFuncs[id];
>+
>+                if (destroyFunc)
>+                {
>+                    destroyFunc (impl);
>+                }
>+
>+                gImpl.myPayloads[id] = NULL;
>             }
>+
>+            closeSharedLib (lib);
>+            gImpl.myPayloadLibraries[id] = NULL;
>         }
>
>        gDefaultPayload = NULL;
>@@ -1827,18 +1856,29 @@ mama_setDefaultPayload (char id)
>
> mama_status
> mama_loadPayloadBridge (mamaPayloadBridge* impl,
>-                         const char*        payloadName)
>+                        const char*        payloadName)
> {
>     return mama_loadPayloadBridgeInternal (impl, payloadName);
> }
>+
>+mama_status
>+mama_loadPayloadBridgeByType (mamaPayloadBridge* impl,
>+                              mamaPayloadType    payload)
>+{
>+    const char* name = mamaPayload_convertToLibString (payload);
>+    return mama_loadPayloadBridgeInternal (impl, name);
>+}
>+
> mama_status
> mama_loadPayloadBridgeInternal  (mamaPayloadBridge* impl,
>                                  const char*        payloadName)
> {
>     char                    bridgeImplName  [256];
>     char                    initFuncName    [256];
>+    char                    destroyFuncName [256];
>     LIB_HANDLE              bridgeLib       = NULL;
>     msgPayload_createImpl   initFunc        = NULL;
>+    msgPayload_destroyImpl  destroyFunc     = NULL;
>     mama_status             status          = MAMA_STATUS_OK;
>     char                    payloadChar       ='\0';
>       void*                                   vp                              = NULL;
>@@ -1865,11 +1905,10 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>         return MAMA_STATUS_NO_BRIDGE_IMPL;
>     }
>
>-    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>-
>     /* Gives a warning - casting from void* to bridge_createImpl func */
>-      vp = loadLibFunc (bridgeLib, initFuncName);
>-      initFunc  = *(msgPayload_createImpl*) &vp;
>+    snprintf (initFuncName, 256, "%sPayload_createImpl",  payloadName);
>+    vp = loadLibFunc (bridgeLib, initFuncName);
>+    initFunc  = *(msgPayload_createImpl*) &vp;
>
>     if (!initFunc)
>     {
>@@ -1887,7 +1926,7 @@ mama_loadPayloadBridgeInternal  (mamaPayloadBridge*
>impl,
>
>     if (MAMA_STATUS_OK != (status = initFunc (impl, &payloadChar)))
>     {
>-       wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return status;
>     }
>@@ -1910,13 +1949,19 @@ mama_loadPayloadBridgeInternal
>(mamaPayloadBridge* impl,
>              "Payload bridge %s already loaded",
>              payloadName);
>
>-         wthread_static_mutex_unlock (&gImpl.myLock);
>+        wthread_static_mutex_unlock (&gImpl.myLock);
>
>         return MAMA_STATUS_OK;
>     }
>
>+    /* Gives a warning - casting from void* to bridge_destroyImpl func */
>+    snprintf (destroyFuncName, 256, "%sPayload_destroyImpl",
>payloadName);
>+    vp = loadLibFunc (bridgeLib, destroyFuncName);
>+    destroyFunc = *(msgPayload_destroyImpl*) &vp;
>+
>     gImpl.myPayloads [(int)payloadChar] = *impl;
>     gImpl.myPayloadLibraries [(int)payloadChar] = bridgeLib;
>+    gImpl.myPayloadDestroyFuncs [(int)payloadChar] = destroyFunc;
>
>     if (!gDefaultPayload)
>     {
>diff --git a/mama/c_cpp/src/c/mama/mama.h b/mama/c_cpp/src/c/mama/mama.h
>index a574f36..07a9115 100644
>--- a/mama/c_cpp/src/c/mama/mama.h
>+++ b/mama/c_cpp/src/c/mama/mama.h
>@@ -133,11 +133,34 @@ extern "C"
>      */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadBridge (mamaBridge* impl, const char*  middleware);
>+    mama_loadBridge (mamaBridge* impl,
>+                     const char* middleware);
>
>+    /**
>+     * Load a payload bridge by its well-known library name string.
>E.g. "wmsg"
>+     * will load the WombatMsg payload, provided it exists on the system.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payloadName The well-known name of the payload library.
>+     */
>+    MAMAExpDLL
>+    extern mama_status
>+    mama_loadPayloadBridge (mamaPayloadBridge* bridge,
>+                            const char*        payloadName);
>+
>+    /**
>+     * Load a payload bridge by using the mamaPayloadType identifier.
>This
>+     * utilises mamaPayload_convertToLibString() in order to retrieve the
>+     * well-known library string for the payload library, and this is
>then
>+     * delegated to mama_loadPayloadBridge.
>+     *
>+     * @param bridge The payload bridge object.
>+     * @param payload The payload enumeration identifier.
>+     */
>     MAMAExpDLL
>     extern mama_status
>-    mama_loadPayloadBridge (mamaPayloadBridge*  bridge,  const char*
>payloadName);
>+    mama_loadPayloadBridgeByType (mamaPayloadBridge* bridge,
>+                                  mamaPayloadType    payload);
>
>     /** Load the bridge specified by middleware string using the path
>specified by the user.
>      * If the bridge has already been loaded then the existing bridge
>instance
>diff --git a/mama/c_cpp/src/c/mama/msg.h b/mama/c_cpp/src/c/mama/msg.h
>index bf0d562..a9c3438 100644
>--- a/mama/c_cpp/src/c/mama/msg.h
>+++ b/mama/c_cpp/src/c/mama/msg.h
>@@ -74,6 +74,36 @@ extern const char*
> mamaPayload_convertToString (mamaPayloadType payloadType);
>
> /**
>+ * Convert a string to a mamaPayloadType.
>+ *
>+ * @param str The str to convert.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromString (const char* str);
>+
>+/**
>+ * Convert a mamaPayloadType value to a well-known library name string
>+ * for that payload.  E.g. MAMA_PAYLOAD_WOMBAT_MSG translates into
>+ * "wmsg" (of which the payload library would be libwmsgimpl.so on *nix).
>+ *
>+ * @param payloadType The payloadType to convert to a library name
>string.
>+ */
>+MAMAExpDLL
>+extern const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType);
>+
>+/**
>+ * Convert a well-known library name to a mamaPayloadType value
>+ * for that payload.  E.g. "wmsg" translates into
>MAMA_PAYLOAD_WOMBAT_MSG.
>+ *
>+ * @param str The library name string to convert to a mamaPayloadType.
>+ */
>+MAMAExpDLL
>+extern mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str);
>+
>+/**
>  * Create a mamaMsg.
>  *
>  * @param msg The location of a mamaMsg where to store the result.
>diff --git a/mama/c_cpp/src/c/msg.c b/mama/c_cpp/src/c/msg.c
>index 286fcad..506c390 100644
>--- a/mama/c_cpp/src/c/msg.c
>+++ b/mama/c_cpp/src/c/msg.c
>@@ -41,13 +41,39 @@
>
> #include "wombat/wincompat.h"
>
>-
> #define MAX_SUBJECT 256
>
>-#define NOFID 0
>-
>-static const int INCLUDE_FIELD_NAME = (1 << 4);
>-static const int INCLUDE_FIELD_ID   = (1 << 2);
>+typedef struct PayloadTable
>+{
>+    mamaPayloadType mType;
>+    const char*     mName;
>+    const char*     mLibrary;
>+}
>+PayloadTable;
>+
>+static
>+const PayloadTable kPayloads [] =
>+{
>+    {MAMA_PAYLOAD_ACTIV,      "activ",      "activmsg"},
>+    {MAMA_PAYLOAD_AVIS,       "AVIS",       "avismsg"},
>+    {MAMA_PAYLOAD_EXEGY,      "EXEGY",      "exegymsg"},
>+    {MAMA_PAYLOAD_FAST,       "FAST",       "fastmsg"},
>+    {MAMA_PAYLOAD_IBMWFO,     "ibmwfo",     "ibmwfomsg"},
>+    {MAMA_PAYLOAD_INRUSH,     "INRUSH",     "inrushmsg"},
>+    {MAMA_PAYLOAD_QPID,       "QPID",       "qpidmsg"},
>+    {MAMA_PAYLOAD_RAI,        "rai",        "raimsg"},
>+    {MAMA_PAYLOAD_SOLACE,     "solacemsg",  "solacemsg"},
>+    {MAMA_PAYLOAD_TIBRV,      "TIBRV",      "tibrvmsg"},
>+    {MAMA_PAYLOAD_TICK42BLP,  "TICK42BLP",  "tick42blpmsg"},
>+    {MAMA_PAYLOAD_TICK42RMDS, "TICK42RMDS", "tick42rmdsmsg"},
>+    {MAMA_PAYLOAD_UMS,        "UMS",        "umsmsg"},
>+    {MAMA_PAYLOAD_V5,         "V5",         "wcachemsg"},
>+    {MAMA_PAYLOAD_VULCAN,     "Vulcan",     "vulcanmsg"},
>+    {MAMA_PAYLOAD_WOMBAT_MSG, "WombatMsg",  "wmsg"},
>+
>+    /* Linear search requires unknown to be at the end-of-table */
>+    {MAMA_PAYLOAD_UNKNOWN,    "unknown",    NULL}
>+};
>
> typedef struct mamaMsgIteratorImpl
> {
>@@ -106,6 +132,66 @@ typedef struct mamaMsgImpl_
> static mama_status
> mamaMsgImpl_destroyLastVectorMsg (mamaMsgImpl *impl);
>
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByType (mamaPayloadType payloadType)
>+{
>+    int k = 0;
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (kPayloads[k].mType == payloadType)
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByName (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mName))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
>+static
>+const PayloadTable*
>+mamaPayloadImpl_lookupTableByLibrary (const char* name)
>+{
>+    int k = 0;
>+
>+    if (!name)
>+    {
>+        return &kPayloads[sizeof(kPayloads)/sizeof(kPayloads[0])-1];
>+    }
>+
>+    for (; kPayloads[k].mType != MAMA_PAYLOAD_UNKNOWN; ++k)
>+    {
>+        if (0 == strcasecmp (name, kPayloads[k].mLibrary))
>+        {
>+            break;
>+        }
>+    }
>+
>+    return &kPayloads[k];
>+}
>+
> /*=================================================================
>   = Public functions - defined in mama/msg.h
>   ================================================================*/
>@@ -166,7 +252,9 @@ mamaMsg_destroy (mamaMsg msg)
>     if (impl->mCurrentField)
>     {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>     }
>+
>     /*Destroy the reusable field object*/
>     if (impl->mCopy)
>     {
>@@ -366,44 +454,29 @@ mamaMsgImpl_getPayload (const mamaMsg msg,
>msgPayload* payload)
> const char*
> mamaPayload_convertToString (mamaPayloadType payloadType)
> {
>-    switch (payloadType)
>-    {
>-        case MAMA_PAYLOAD_SOLACE:
>-            return "solacemsg";
>-        case MAMA_PAYLOAD_V5:
>-            return "V5";
>-        case MAMA_PAYLOAD_AVIS:
>-            return "AVIS";
>-        case MAMA_PAYLOAD_TICK42BLP:
>-            return "TICK42BLP";
>-        case MAMA_PAYLOAD_FAST:
>-            return "FAST";
>-        case MAMA_PAYLOAD_RAI:
>-            return "rai";
>-        case MAMA_PAYLOAD_UMS:
>-            return "UMS";
>-        case MAMA_PAYLOAD_TICK42RMDS:
>-            return "TICK42RMDS";
>-        case MAMA_PAYLOAD_QPID:
>-            return "QPID";
>-        case MAMA_PAYLOAD_TIBRV:
>-            return "TIBRV";
>-        case MAMA_PAYLOAD_IBMWFO:
>-            return "ibmwfo";
>-        case MAMA_PAYLOAD_ACTIV:
>-            return "activ";
>-        case MAMA_PAYLOAD_VULCAN:
>-            return "Vulcan";
>-        case MAMA_PAYLOAD_WOMBAT_MSG:
>-            return "WombatMsg";
>-        case MAMA_PAYLOAD_EXEGY:
>-            return "EXEGY";
>-        case MAMA_PAYLOAD_INRUSH:
>-            return "INRUSH";
>-
>-        default:
>-            return "unknown";
>-    }
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mName;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByName (str);
>+    return table->mType;
>+}
>+
>+const char*
>+mamaPayload_convertToLibString (mamaPayloadType payloadType)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByType
>(payloadType);
>+    return table->mLibrary;
>+}
>+
>+mamaPayloadType
>+mamaPayload_convertFromLibString (const char* str)
>+{
>+    const PayloadTable* table = mamaPayloadImpl_lookupTableByLibrary
>(str);
>+    return table->mType;
> }
>
> mama_status
>@@ -3481,7 +3554,11 @@ mama_status mamaMsgIterator_destroy
>(mamaMsgIterator iterator)
>     }
>
>     if (impl->mCurrentField)
>+    {
>         mamaMsgField_destroy (impl->mCurrentField);
>+        impl->mCurrentField = NULL;
>+    }
>+
>     free (impl);
>     return MAMA_STATUS_OK;
> }
>diff --git a/mama/c_cpp/src/c/payloadbridge.h
>b/mama/c_cpp/src/c/payloadbridge.h
>index 703c1a9..6ad7108 100644
>--- a/mama/c_cpp/src/c/payloadbridge.h
>+++ b/mama/c_cpp/src/c/payloadbridge.h
>@@ -422,8 +422,12 @@ while(0)
>                       \
> typedef mama_status
> (*msgPayload_createImpl)      (mamaPayloadBridge* result, char*
>identifier);
>
>+typedef mama_status
>+(*msgPayload_destroyImpl)     (mamaPayloadBridge result);
>+
> typedef mamaPayloadType
> (*msgPayload_getType)         (void);
>+
> /*===================================================================
>  =              msgPayload bridge function pointers                 =
>  ====================================================================*/
>diff --git a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>index ed18a64..1e14227 100644
>--- a/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>+++ b/mama/c_cpp/src/gunittest/c/payloadmiddlewareidtest.cpp
>@@ -20,14 +20,9 @@ protected:
>
>    virtual void TearDown(void);
>
>-   typedef std::pair <mamaPayloadType, std::string> MamaPayloadPair;
>-   typedef std::map  <mamaPayloadType, std::string> MamaPayloadMapType;
>-
>-   typedef std::pair <mamaMiddleware,  std::string> MamaMiddlewarePair;
>-   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>-
>-   MamaPayloadMapType     payloadTestData;
>-   MamaMiddlewareMapType  middlewareTestData;
>+   typedef std::pair <std::string,     std::string>    PayloadNameLib;
>+   typedef std::map  <mamaPayloadType, PayloadNameLib>
>MamaPayloadMapType;
>+   typedef std::map  <mamaMiddleware,  std::string>
>MamaMiddlewareMapType;
>
> private:
>    void CreateTestData();
>@@ -55,100 +50,81 @@ MamaEnumTestsC::TearDown()
> void
> MamaEnumTestsC::CreateTestData()
> {
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_V5,
>"V5"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_AVIS,
>"AVIS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42BLP,
>"TICK42BLP"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_FAST,
>"FAST"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_RAI,
>"rai"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UMS,
>"UMS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TICK42RMDS,
>"TICK42RMDS"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_QPID,
>"QPID"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_TIBRV,
>"TIBRV"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_IBMWFO,
>"ibmwfo"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_ACTIV,
>"activ"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_VULCAN,
>"Vulcan"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_WOMBAT_MSG,
>"WombatMsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_EXEGY,
>"EXEGY"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_SOLACE,
>"solacemsg"));
>-    payloadTestData.insert (MamaPayloadPair (MAMA_PAYLOAD_UNKNOWN,
>"unknown"));
>-
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_WMW,
>     "wmw"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_LBM,
>     "lbm"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TIBRV,     "tibrv"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_AVIS,
>     "AVIS"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42BLP, "tick42blp"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_SOLACE,    "SOLACE"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_RAI,
>     "rai"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_QPID,
>     "QPID"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_EXEGY,     "exegy"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_IBMWFO,    "ibmwfo"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_ACTIV,     "activ"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_TICK42RMDS, "tick42rmds"));
>-    middlewareTestData.insert (MamaMiddlewarePair (MAMA_MIDDLEWARE_UMS,
>     "ums"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_VULCAN,    "vulcan"));
>-    middlewareTestData.insert (MamaMiddlewarePair
>(MAMA_MIDDLEWARE_UNKNOWN,   "unknown"));
> }
>
>-
>-TEST_F (MamaEnumTestsC, testPayloadConvertToString)
>+TEST_F (MamaEnumTestsC, testPayloadStringConversions)
> {
>+    MamaPayloadMapType testData;
>+    testData[MAMA_PAYLOAD_ACTIV]      = PayloadNameLib ("activ",
>"activmsg");
>+    testData[MAMA_PAYLOAD_AVIS]       = PayloadNameLib ("AVIS",
>"avismsg");
>+    testData[MAMA_PAYLOAD_EXEGY]      = PayloadNameLib ("EXEGY",
>"exegymsg");
>+    testData[MAMA_PAYLOAD_FAST]       = PayloadNameLib ("FAST",
>"fastmsg");
>+    testData[MAMA_PAYLOAD_IBMWFO]     = PayloadNameLib ("ibmwfo",
>"ibmwfomsg");
>+    testData[MAMA_PAYLOAD_QPID]       = PayloadNameLib ("QPID",
>"qpidmsg");
>+    testData[MAMA_PAYLOAD_RAI]        = PayloadNameLib ("rai",
>"raimsg");
>+    testData[MAMA_PAYLOAD_SOLACE]     = PayloadNameLib ("solacemsg",
>"solacemsg");
>+    testData[MAMA_PAYLOAD_TIBRV]      = PayloadNameLib ("TIBRV",
>"tibrvmsg");
>+    testData[MAMA_PAYLOAD_TICK42BLP]  = PayloadNameLib ("TICK42BLP",
>"tick42blpmsg");
>+    testData[MAMA_PAYLOAD_TICK42RMDS] = PayloadNameLib ("TICK42RMDS",
>"tick42rmdsmsg");
>+    testData[MAMA_PAYLOAD_UMS]        = PayloadNameLib ("UMS",
>"umsmsg");
>+    testData[MAMA_PAYLOAD_VULCAN]     = PayloadNameLib ("Vulcan",
>"vulcanmsg");
>+    testData[MAMA_PAYLOAD_WOMBAT_MSG] = PayloadNameLib ("WombatMsg",
>"wmsg");
>+    testData[MAMA_PAYLOAD_V5]         = PayloadNameLib ("V5",
>"wcachemsg");
>+    testData[MAMA_PAYLOAD_UNKNOWN]    = PayloadNameLib ("unknown",
>"");
>+
>     MamaPayloadMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = payloadTestData.begin(); itr != payloadTestData.end();
>++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaPayloadType payload   = (*itr).first;
>-       std::string     expected  = (*itr).second;
>-
>-       std::string actual = mamaPayload_convertToString (payload);
>-
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       mamaPayloadType payload = itr->first;
>+       PayloadNameLib& namelib = itr->second;
>+       const char*     name    = namelib.first.c_str();
>+       const char*     library = namelib.second.c_str();
>+
>+       // Override for unknown, mama returns NULL for unknown library
>(so that
>+       // it prevents libraries from trying to load libmamaunknownmsg).
>+       if (MAMA_PAYLOAD_UNKNOWN == payload)
>+       {
>+           library = NULL;
>+       }
>+
>+       // Check name/payload conversion
>+       EXPECT_STREQ(name, mamaPayload_convertToString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromString
>(name));
>+
>+       // Check library/payload conversion
>+       EXPECT_STREQ(library, mamaPayload_convertToLibString (payload));
>+       EXPECT_EQ((char)payload, (char)mamaPayload_convertFromLibString
>(library));
>     }
>-
>-    ASSERT_EQ( passed, true );
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertToString)
>+TEST_F (MamaEnumTestsC, testMiddlewareStringConversions)
> {
>+    MamaMiddlewareMapType testData;
>+    testData[MAMA_MIDDLEWARE_WMW]        = "wmw";
>+    testData[MAMA_MIDDLEWARE_LBM]        = "lbm";
>+    testData[MAMA_MIDDLEWARE_TIBRV]      = "tibrv";
>+    testData[MAMA_MIDDLEWARE_AVIS]       = "AVIS";
>+    testData[MAMA_MIDDLEWARE_TICK42BLP]  = "tick42blp";
>+    testData[MAMA_MIDDLEWARE_SOLACE]     = "SOLACE";
>+    testData[MAMA_MIDDLEWARE_RAI]        = "rai";
>+    testData[MAMA_MIDDLEWARE_QPID]       = "QPID";
>+    testData[MAMA_MIDDLEWARE_EXEGY]      = "exegy";
>+    testData[MAMA_MIDDLEWARE_IBMWFO]     = "ibmwfo";
>+    testData[MAMA_MIDDLEWARE_ACTIV]      = "activ";
>+    testData[MAMA_MIDDLEWARE_TICK42RMDS] = "tick42rmds";
>+    testData[MAMA_MIDDLEWARE_UMS]        = "ums";
>+    testData[MAMA_MIDDLEWARE_VULCAN]     = "vulcan";
>+    testData[MAMA_MIDDLEWARE_UNKNOWN]    = "unknown";
>     MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>
>-    for(itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); ++itr) {
>+    for(itr = testData.begin(); itr != testData.end(); ++itr) {
>
>-       mamaMiddleware middleware = (*itr).first;
>-       std::string    expected   = (*itr).second;
>-
>-       std::string actual = mamaMiddleware_convertToString (middleware);
>+       mamaMiddleware middleware = itr->first;
>+       const char*    name       = itr->second.c_str();
>
>-       EXPECT_STREQ(actual.c_str(), expected.c_str());
>-
>-       if (actual != expected)
>-           passed = false;
>+       EXPECT_STREQ(name, mamaMiddleware_convertToString (middleware));
>+       EXPECT_EQ((char)middleware,
>(char)mamaMiddleware_convertFromString (name));
>     }
>-
>-    ASSERT_EQ (passed, true);
> }
>
>-TEST_F (MamaEnumTestsC, testMiddlewareConvertFromString)
>-{
>-    MamaMiddlewareMapType::iterator itr;
>-    bool passed = true;
>-
>-    for (itr = middlewareTestData.begin(); itr !=
>middlewareTestData.end(); itr++) {
>-
>-        mamaMiddleware expected    = (*itr).first;
>-        std::string    middleware  = (*itr).second;
>-
>-        mamaMiddleware actual      = mamaMiddleware_convertFromString
>(middleware.c_str());
>-
>-        EXPECT_EQ (actual, expected);
>-
>-        if (actual != expected)
>-            passed = false;
>-    }
>-
>-    ASSERT_EQ (passed, true);
>-}
>

________________________________________________________

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.
________________________________________________________



 

--

Lee Skillen

 

Vulcan Financial Technologies

51 Malone Road, Belfast, BT9 6RY

 

Web:     www.vulcanft.com 



 

--

Lee Skillen

 

Vulcan Financial Technologies

51 Malone Road, Belfast, BT9 6RY

 

Office:  +44 (0)28 95 817888

Mobile:  +44 (0)78 41 425152

Web:     www.vulcanft.com 


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.



This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange Group, Inc. (ICE), NYSE Euronext or any of their subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.