Date   

Running latest OpenMAMA MamaResourcePool changes with Microsoft's Vcpkg

Frank Quinn
 

Hi Folks,

 

The following is to make testing the most recent changes easier particularly on windows… I have my own fork of vcpkg which points to the latest mama resource pool code.

 

Run this git command to grab a cut of my fork of vcpkg pointing to the latest openmama resource pool changes from my working branch

 

    git clone -b openmama-resource-pool-cut https://github.com/fquinner/vcpkg.git vcpkg-openmama

 

Then set up vcpkg:

 

    cd vcpkg-openmama

    .\bootstrap-vcpkg.bat

 

And install OpenMAMA into this environment:

 

    vcpkg install openmama:x64-windows

 

Then use Visual Studio / other IDE to open each tutorial from https://github.com/finos/OpenMAMA/tree/next/tutorials/cpp as a cmake project. In your cmake configuration, then set CMAKE_TOOLCHAIN_FILE=\path\to\vcpkg-openmama\scripts\buildsystems\vcpkg.cmake to let your development environment pick up openmama via Cmake’s find_package with no further modifications / environment variables.

 

Cheers,

Frank

 

Frank Quinn

Cascadium

T: +44 (0) 28 8678 8015

E: fquinn@...

W: https://cascadium.io

 


Re: OpenMAMA Resource Pool merged into development branch

Frank Quinn
 

Hi Folks,

 

Bumping this up to the mailing list again for review.

 

Again, any feedback let me know either via email / issue or Gitter.

 

Cheers,

Frank

 

Frank Quinn, Cascadium | +44 (0) 28 8678 8015 | https://cascadium.io

 

From: Frank Quinn
Sent: 16 August 2022 15:16
To: openmama-dev@...
Subject: OpenMAMA Resource Pool merged into development branch

 

Hi Folks,

I’m happy to report that we have the OpenMAMA Resource Pool ready go to in the "next" branch.

The C++ MamaResourcePool even now plays nicely with smart pointers now.

The implementation includes C, C++, Java and C# bindings and should be ready for testing.

Documentation and tutorials for new API

https://openmama.finos.org/openmama_developer_guide.html#resource-pool-work-in-progress

https://openmama.finos.org/tutorial_mama_resource_pool.html

Please feel free to give it a go and raise issues if you find any problems! Any quick questions, swing by the gitter page.

Cheers,

Frank

Frank QuinnCascadium | +44 (0) 28 8678 8015 | https://cascadium.io


OpenMAMA Resource Pool merged into development branch

Frank Quinn
 

Hi Folks,

I’m happy to report that we have the OpenMAMA Resource Pool ready go to in the "next" branch.

The C++ MamaResourcePool even now plays nicely with smart pointers now.

The implementation includes C, C++, Java and C# bindings and should be ready for testing.

Documentation and tutorials for new API

https://openmama.finos.org/openmama_developer_guide.html#resource-pool-work-in-progress

https://openmama.finos.org/tutorial_mama_resource_pool.html

Please feel free to give it a go and raise issues if you find any problems! Any quick questions, swing by the gitter page.

Cheers,

Frank

Frank QuinnCascadium | +44 (0) 28 8678 8015 | https://cascadium.io


Documentation Formats on Website Updated

Frank Quinn
 

Hi Folks,

 

FYI there’s a bunch of updated documentation on the website including a fully integrated / amalgamated version of the developer guides as well as doxygen documentation which is no longer stuffed into a little iframe.

 

It’s now live at:

 

https://openmama.finos.org/openmama_developer_guide.html

 

And:

 

https://openmama.finos.org/openmama_api_reference_docs.html

 

Note the developer guide is still to be subject to a content review – this is more of a port / fix of some of things which were “already there” rather than an overall content refresh.

 

More content will be available too in the coming weeks / months.

 

Enjoy!

 

Cheers,

Frank

 

Frank Quinn

Cascadium

T: +44 (0) 28 8678 8015

E: fquinn@...

W: https://cascadium.io

 


Linux Supported Platforms

Philip Preston
 

ahead


Linux Supported Platforms

Frank Quinn
 

Hi folks,

I have recently been giving some thought to our currently supported Linux platforms which we package. I would be of the mind that:

1. We should drop Fedora packaging. No-one seems to use it and it moves very very quickly and breaks things regularly. The goal behind this was always to remain ahead of upcoming breaking RH changes but we now have CentOS Stream for that. Which brings me to...
2. We should add CentOS stream 8 and 9
3. We should drop Ubuntu 16.04 LTS and adopt Ubuntu 22.04 LTS
4. We should drop CentOS 6

Doing the above would also (finally) pave the way for decent C++11 support since no supported platform compilers remain which don't have good coverage of it.

Any thoughts welcome though but that's where my head is at at the moment.

Cheers,
Frank


Re: CentOS 6 / RHEL 6 Support (Reply Needed)

Frank Quinn
 

Hi Folks,

 

Now that CentOS has been EOL for 16 months, could I ask for feedback again on the below please? 1, 2 or 3?

 

Cheers,

Frank

 

Frank Quinn, Cascadium | +44 (0) 28 8678 8015 | https://cascadium.io

 

From: Openmama-users@... <Openmama-users@...> On Behalf Of Frank Quinn via lists.openmama.org
Sent: 20 November 2020 09:30
To: openmama-dev@...; openmama-users@...
Subject: [Openmama-users] CentOS 6 / RHEL 6 Support (Reply Needed)

 

Hi Folks,

 

CentOS 6 is reaching its main EOL on the 30th November (10 days time), so I’d like to ask the community where they are in respect to migration.

 

Please feel free to reply to me either directly or via list with one of the below:

 

  1. CentOS 6? How quaint. Kill it with fire.
  2. We are moving off CentOS 6 and expect to be migrated away from it before the next OpenMAMA release in Summer.
  3. I think we’d need CentOS support for at least another year or so until all our systems migrate.

 

Cheers,

Frank

 

Frank Quinn

Cascadium

T: +44 (0) 28 8678 8015

E: fquinn@...

W: http://cascadium.io

 


Re: OpenMAMA Resource Pool API RFC

Frank Quinn
 

Thanks your consideration folks, this RFC is now considered approved and work will begin in the coming weeks.

 

Cheers,

Frank

 

Frank Quinn, Cascadium | +44 (0) 28 8678 8015 | https://cascadium.io

 

From: Openmama-dev@... <Openmama-dev@...> On Behalf Of Frank Quinn via lists.openmama.org
Sent: 14 January 2022 12:33
To: Openmama-dev@...
Subject: [Openmama-dev] OpenMAMA Resource Pool API RFC

 

Hi Folks,

 

I have just opened a new RFC to cover the OpenMAMA Resource Pool originally discussed on the mailing list to try and make the OpenMAMA API more accessible for new users without sacrificing performance or functionality:

 

https://openmama.finos.org/openmama_rfc_mama_resource_pool.html

 

As per our RFC process, we will leave this open for a period of 2 weeks, after which point if there are no major concerns, the RFC design will be considered approved, and work can begin on the recommended option in the document.

 

For details on how to provide constructive RFC feedback, please refer to https://openmama.github.io/openmama_rfc_process.html#how-to-provide-rfc-feedback

 

I’ll be on Gitter for any quick informal questions anyone may have throughout as well.

 

Cheers,

Frank

 

Frank Quinn

Cascadium

T: +44 (0) 28 8678 8015

E: fquinn@...

W: https://cascadium.io

 


OpenMAMA Resource Pool API RFC

Frank Quinn
 

Hi Folks,

 

I have just opened a new RFC to cover the OpenMAMA Resource Pool originally discussed on the mailing list to try and make the OpenMAMA API more accessible for new users without sacrificing performance or functionality:

 

https://openmama.finos.org/openmama_rfc_mama_resource_pool.html

 

As per our RFC process, we will leave this open for a period of 2 weeks, after which point if there are no major concerns, the RFC design will be considered approved, and work can begin on the recommended option in the document.

 

For details on how to provide constructive RFC feedback, please refer to https://openmama.github.io/openmama_rfc_process.html#how-to-provide-rfc-feedback

 

I’ll be on Gitter for any quick informal questions anyone may have throughout as well.

 

Cheers,

Frank

 

Frank Quinn

Cascadium

T: +44 (0) 28 8678 8015

E: fquinn@...

W: https://cascadium.io

 


Re: Subscription Create Behaviour

Frank Quinn
 

Hi Mike,

 

Subscription activations may be created upfront before mama_start if not using the throttle yes.

 

The behaviour you have identified is similar to behaviour outlined in the OpenMAMA Developer’s guide in the MAMA Subscription Callbacks section where it is possible to receive an onMsg before the onCreate has even been fired. It’s the same sort of principle in both cases where the thread performing the setup differs from the thread performing the dispatch. Note I expect the behaviour you have observed will only happen if you’re (quite rightly) not using the default queue for your subscriptions.

 

If you want to prevent this behaviour, adding your own MAMA Queue event to initialized all unthrottled subscriptions when the default queue unblocks could be an option.

 

Cheers,

Frank

Frank Quinn, Cascadium | +44 (0) 28 8678 8015 | http://cascadium.io

 

From: Openmama-dev@... <Openmama-dev@...> On Behalf Of Slade, Michael J via lists.openmama.org
Sent: 17 May 2021 11:02
To: openmama-dev@...
Subject: [Openmama-dev] Subscription Create Behaviour

 

Hi OpenMAMA Dev,

 

I wanted to clarify the expected behavior surrounding creation of subscriptions during application startup – should subscription activations always be executed by the thread dispatching events off the throttle queue?

 

Taking mamalistenc as an example, subscriptions can be created before we start dispatching on the default event queue. If the throttle rate is non-zero, the subscriptions’ activations are added to the throttle queue after the set interval has expired, but will not be executed until we start dispatching on the queue when we call mama_start.

 

The other behavior we see is if the throttle is disabled (the rate is set to <=0). The subscriptions’ activations are executed immediately on the calling thread – ignoring the queue - before the mama_start call due to the below code snippet. This results in the subscription being created on the transport and messages being received from the middleware before the mama_start call. Is this expected/allowed behavior?

 

Throttle.c:

if (self->mRate <= 0.0)

{

    /* throttle is not in use, execute immediately */

    mama_log (MAMA_LOG_LEVEL_FINEST, "wombatThrottle_dispatch (): "

              "no throttle; triggering action.");

    actionCb (closure1, closure2);

}

 

Thanks in advance,

Mike

This message is confidential and subject to terms at: https://www.jpmorgan.com/emaildisclaimer including on confidential, privileged or legal entity information, malicious content and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.


OpenMAMA OMNM 1.0.0 Released

Frank Quinn
 

Hi Folks,

 

It has been a while since the last release but we’re pleased to announce the release of OpenMAMA OMNM 1.0.0. This release includes:

 

  • Complete interface compatibility with OpenMAMA
  • Complete unit test coverage for all official OpenMAMA unit tests
  • Platform support for all OpenMAMA supported platforms including Windows and OSX
  • A set of methods to support extending omnm for use in other payload bridges with reduced code required:
    • omnmmsgPayloadImpl_[gs]etExtenderClosure to allow the extender
      to attach their own internal object to the omnm payload
    • omnmmsgPayloadImpl_updateVectorMsgPayload to allow the extender
      to update vectors of messages without needing an intermediate
      mamaMsg for each submsg.

Release artifacts can be found here:

https://github.com/cascadium/OpenMAMA-omnm/releases

Or by running:

yum / apt upgrade openmama-omnm

From the OpenMAMA thirdparty repository hosted by our friends at cloudsmith:

https://cloudsmith.io/~openmama/repos/openmama-thirdparty/packages/

The first implementation based on this base bridge will be released soon too and adds support for msgpack.

 

Cheers,

Frank

 

Frank Quinn

Cascadium

T: +44 (0) 28 8678 8015

E: fquinn@...

W: http://cascadium.io

 


Subscription Create Behaviour

Slade, Michael J
 

Hi OpenMAMA Dev,

 

I wanted to clarify the expected behavior surrounding creation of subscriptions during application startup – should subscription activations always be executed by the thread dispatching events off the throttle queue?

 

Taking mamalistenc as an example, subscriptions can be created before we start dispatching on the default event queue. If the throttle rate is non-zero, the subscriptions’ activations are added to the throttle queue after the set interval has expired, but will not be executed until we start dispatching on the queue when we call mama_start.

 

The other behavior we see is if the throttle is disabled (the rate is set to <=0). The subscriptions’ activations are executed immediately on the calling thread – ignoring the queue - before the mama_start call due to the below code snippet. This results in the subscription being created on the transport and messages being received from the middleware before the mama_start call. Is this expected/allowed behavior?

 

Throttle.c:

if (self->mRate <= 0.0)

{

    /* throttle is not in use, execute immediately */

    mama_log (MAMA_LOG_LEVEL_FINEST, "wombatThrottle_dispatch (): "

              "no throttle; triggering action.");

    actionCb (closure1, closure2);

}

 

Thanks in advance,

Mike

This message is confidential and subject to terms at: https://www.jpmorgan.com/emaildisclaimer including on confidential, privileged or legal entity information, malicious content and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.


OpenMAMA "top level" Common Headers

Frank Quinn
 

Hi Folks,

 

With more and more packaging options being made available to OpenMAMA, it was only a matter of time before the fact that our build system includes top level header files was to become an issue.

 

This was raised by our friends at Microsoft / vcpkg when they discovered that it conflicted with one of their headers.

 

I have suggested a workaround https://github.com/Cheney-W/vcpkg/pull/1 which was ultimately merged upstream, but it involves moving these trailing headers into the “wombat” subdirectory. So that’s:

 

  • destroyhandle.h
  • platform.h
  • list.h
  • lookup2.h
  • property.h
  • timers.h
  • wlock.h

 

Which will all shortly be moved into the wombat subdirectory through our build system when installing.

 

If you make direct use of these headers, to fix this in your code if you use these headers directly, you may simply add $MAMA_ROOT/include/wombat to your include paths (or better still, update your #include statements to use <wombat/>). The former is something you should be able to do now before the change lands so you are forwards compatible with it.

 

This is planned to be included in the next upcoming OpenMAMA release (currently 6.3.2).

 

If anyone desperately wants a configuration option to allow the legacy header location, that can also be considered, though since we’re trying to get included in as many upstream packing targets as possible, it will not be the default for our packaging going forward.

 

Cheers,

Frank

 

Frank Quinn

Cascadium

T: +44 (0) 28 8678 8015

E: fquinn@...

W: http://cascadium.io

 


Re: A new "MamaResourcePool" API for OpenMAMA?

Bill Torpey
 

Hi Frank:

I read with interest your latest email, and certainly agree that there are ways to make OpenMAMA both easier to use (and harder to mis-use!), and more accessible to new users. We've been working on these issues since NYFIX was initially acquired by NYSE back in 2009, and have some experience (and code!) to share as a result.

For us, the biggest problem in moving our middleware stack to Mama/OpenMAMA back in the day (from Tibco Rendezvous), was the relatively poor support for setup and teardown of Mama objects. This wasn't totally surprising, as the typical use case for Mama at the time consisted of relatively long-lived publishers, with subscribers that would come and go during the day. However, in our case the subscribers (servers) tend to be the long-lived components and the publishers (clients) tended to come and go. We had a lot of problems coordinating object lifetimes, especially in a multi-threaded environment, such that we avoided memory leaks, or even worse errors such as use-after-free.

At the time, we worked with the Mama development team to create a wrapper layer called "MME" (for MAMA Managed Environment) which would manage the lifetimes of Mama event objects and remove the biggest problem in Mama: the requirement that object destruction occur only on the event dispatching thread. At the time, we hoped that MME would become part of Mama proper, but that never happened and the rights to the code were transferred to NYFIX when NYSE divested both NYFIX and SR Labs. NYFIX released the code as open-source concurrent with our release of OZ at https://github.com/nyfix/MME.

We've been using MME in the NYFIX Marketplace since approximately 2010; first with the commercial transport from SR/Vela, and for the past year and change with OZ (https://github.com/nyfix/OZ), the open-source transport that we created based on ZeroMQ.

We would be interested in working with the OpenMAMA community to extend and improve MME, and suggest that it answers several of the issues you mentioned.

One thing that MME does not address, however, is overall ease-of-use, and specifically the steep learning-curve associated with the Mama API. For that, we'd like to suggest that the community take a look at the example code included with OZ (https://github.com/nyfix/OZ/blob/master/examples/Readme.md). The example code encapsulates a lot of the "tricky bits" associated with the Mama API, enabling someone to get started quickly -- for example, see https://github.com/nyfix/OZ/blob/master/examples/sub.cpp.

Full disclosure: the OZ examples are exclusively C++ -- while the OZ transport itself is written in C, we prefer C++ for application-level code. The OZ examples do, however, work with other transports, specifically the (now deprecated) Qpid transport.

Both MME and OZ are tried and tested in production use, which is potentially a significant advantage. In any event, we'd be delighted to engage with the OpenMAMA community around either or both of these components to see how they could possibly address some or all of the important issues you raise. We hope the community will take the time to explore both MME and OZ, and we're very interested in any feedback.

Best Regards,

Bill Torpey


A new "MamaResourcePool" API for OpenMAMA?

Frank Quinn
 

Hi Folks,

 

We are currently considering options for trying to make the OpenMAMA API easier to work with and more accessible for new users.

 

When I look at the OpenMAMA API's strengths and weaknesses, the setup and management of the subscriptions themselves is one of the more difficult things to manage especially with a new user. These are things which we seasoned users would take for granted but I think new users would struggle with.

 

That is why I'm suggesting a new "MamaResourcePool" api.

 

The idea is to manage OpenMAMA resources via OpenMAMA itself rather than depending on the user to manage complex state, setup, shutdown etc. I'm taking the approach of starting with what I (pretending to be a new OpenMAMA user) would like to see as opposed to things which would be technically easier to do etc.

 

So taking for example a new subscription, this is the entire code I'm proposing (in C here but should translate easily to other languages) should be required:

 

// Set up the event handlers for OpenMAMA

mamaMsgCallbacks callbacks;

memset(&callbacks, 0, sizeof(callbacks));

callbacks.onMsg = subscriptionOnMsg;

callbacks.onCreate = subscriptionOnCreate;

callbacks.onError = subscriptionOnError;

 

// Use a new "resource pool"

mamaResourcePool resourcePool;

const char* poolName = NULL;

const void* closure = NULL;

 

// Create the resource pool, allocating queues, defaults etc from configuration

mamaResourcePool_create(&resourcePool, poolName);

 

// Actually... subscribe to something via something convenient like a URI

mamaResourcePool_createSubscriptionFromUri(&subscription, "bridge://transport/source/topic", &callbacks, closure);

 

// Block

mama_start();

 

And then you would start receiving events. No transport creation, source creation, queue creation, encouraging users use default queue etc.

 

I’d also suggest that the callbacks onCreate and onError should be optional, with the default implementation simply logging those events.

 

I foresee similar with lambdas etc. The idea is that mamaResourcePool will:

 

·         When it is created:

o    Create a number of event queues for subscriptions (which could be configurable in properties based on the poolName)

o    Load all payload and middleware bridges found on the PATH / LD_LIBRARY_PATH etc

o    Call mama_open() (if not already open)

o    Create and manage internal stores of all managed subscriptions, transports, sources etc

·         Then when a resource is created inside:

o    Find / create transport bridge

o    Find / create transport

o    Find / create source

o    Find / create subscription (e.g. multiple callbacks for same subscription)

o    Find / create anything which is required to acquire the requested resource

·         Be convenient and usable, but still flexible (it would return existing OpenMama objects so they can be manipulated)

·         Take the heavy lifting and thread safety pain away from the application developer

·         Defer default "is this basic sub" or "is this market data sub" to the bridge, configuration or as an API call in the resource pool itself

 

My guess is that many of you may already have an API that does this in each of the provided languages… but newcomers won't have that. This would vastly reduce the complexity and error checking required for a new user, as well as remove the expectation around deciding “which subscription API to use” from the application developer.

 

If there is any feedback to this approach, or in particular if anyone has recently worked with a new user coming online with OpenMAMA please let us know or pop into Gitter to discuss!

 

Cheers,

Frank

 

Frank Quinn

Cascadium

T: +44 (0) 28 8678 8015

E: fquinn@...

W: http://cascadium.io

 


OpenMAMA Now Available on Homebrew (Mac / Linux)

Frank Quinn
 

That’s right Mac users – OpenMAMA has been approved and is now included in homebrew’s set of packages.

 

So to install OpenMAMA on a Mac, simply

 

brew update

brew install openmama

 

…and yes this will also work on the new Apple M1 machines.

 

This also has been verified as working with linuxbrew (if anyone uses that).

 

Any issues please raise an issue on the OpenMAMA github page!

 

Cheers,

Frank

 

Frank Quinn

Cascadium

T: +44 (0) 28 8678 8015

E: fquinn@...

W: http://cascadium.io

 


Re: The future of CentOS and OpenMAMA

Bill Torpey
 

Hi Frank:

At NYFIX we not too long ago moved from RH to CentOS 7 for our dev and prod installs, and assumed that CentOS 8 would be our eventual upgrade path.  

It’s way too early to say how things shake out, with the exception that we will certainly NOT be moving to CentOS 8 Stream.

Personally I’ve moved my day-to-day development from CentOS to Ubuntu, and I’m quite happy with it.  I now only use CentOS to regression test before pushing changes.

Hope this helps.

Regards,

Bill



On Feb 24, 2021, at 12:34 PM, Frank Quinn <fquinn@...> wrote:

Hi Folks,
 
I’m curious about the community feeling towards the future of CentOS. I’d appreciate a reply with one or a selection of the following for those of you running Red Hat derivative distributions:
 
  1. We will be clinging onto CentOS 7 for dear life since CentOS 8 will be EOL 2021 but CentOS 7 won’t be EOL until 2024
  2. We will be making the jump to CentOS Stream
  3. We will go with one of the CentOS clones
  4. We will bide our time and wait to see which CentOS clone comes out on top
  5. This doesn’t really affect us – we use RHEL everywhere
 
Cheers,
Frank
 
Frank Quinn
Cascadium
T: +44 (0) 28 8678 8015
 


The future of CentOS and OpenMAMA

Frank Quinn
 

Hi Folks,

 

I’m curious about the community feeling towards the future of CentOS. I’d appreciate a reply with one or a selection of the following for those of you running Red Hat derivative distributions:

 

  1. We will be clinging onto CentOS 7 for dear life since CentOS 8 will be EOL 2021 but CentOS 7 won’t be EOL until 2024
  2. We will be making the jump to CentOS Stream
  3. We will go with one of the CentOS clones
  4. We will bide our time and wait to see which CentOS clone comes out on top
  5. This doesn’t really affect us – we use RHEL everywhere

 

Cheers,

Frank

 

Frank Quinn

Cascadium

T: +44 (0) 28 8678 8015

E: fquinn@...

W: http://cascadium.io

 


Goodbye Appveyor, Hello Github Actions

Frank Quinn
 

Hi folks,

I am in the final stages of the effort to migrate from Appveyor to Github Actions. This will reduce the overall build time down from 5 hours to 40 minutes across 14 different development environments including new support for OSX in continuous integration. It also provides much richer options around bringing our own compute should we need to with better support for parallel jobs.

These changes will also (finally) retire the Scons build system so if you are still using it, please use cmake instead as documented here:

https://openmama.finos.org/openmama_build_instructions.html

This will be dropping into the "next" branch at some point over the weekend if anyone finds any issues please let me know!

Cheers,
Frank


CentOS 6 / RHEL 6 Support (Reply Needed)

Frank Quinn
 

Hi Folks,

 

CentOS 6 is reaching its main EOL on the 30th November (10 days time), so I’d like to ask the community where they are in respect to migration.

 

Please feel free to reply to me either directly or via list with one of the below:

 

  1. CentOS 6? How quaint. Kill it with fire.
  2. We are moving off CentOS 6 and expect to be migrated away from it before the next OpenMAMA release in Spring.
  3. I think we’d need CentOS support for at least another year or so until all our systems migrate.

 

Cheers,

Frank

 

Frank Quinn

Cascadium

T: +44 (0) 28 8678 8015

E: fquinn@...

W: http://cascadium.io

 

1 - 20 of 2314