|
OpenMAMA Now Available on Homebrew (Mac / Linux)
Frank Quinn
That’s right Mac users – OpenMAMA has been approved and is now included in homebrew’s set of packages.
So to install OpenMAMA on a Mac, simply
brew update brew install openmama
…and yes this will also work on the new Apple M1 machines.
This also has been verified as working with linuxbrew (if anyone uses that).
Any issues please raise an issue on the OpenMAMA github page!
Cheers, Frank
Frank Quinn Cascadium T: +44 (0) 28 8678 8015 E: fquinn@...
|
|
|
|
CentOS 6 / RHEL 6 Support (Reply Needed)
Frank Quinn
Hi Folks,
CentOS 6 is reaching its main EOL on the 30th November (10 days time), so I’d like to ask the community where they are in respect to migration.
Please feel free to reply to me either directly or via list with one of the below:
Cheers, Frank
Frank Quinn Cascadium T: +44 (0) 28 8678 8015 E: fquinn@...
|
|
|
|
Announcing OZ: a production-quality, open-source transport for OpenMAMA using ZeroMQ
Everyone on this mailing list is already well acquainted with OpenMAMA's awesome-ness, but I want to let you know about something that makes OpenMAMA even more awesome. One thing that has always been a trouble spot for OpenMAMA is the lack of a reliable, high-performance, open-source transport library. The AVIS bridge was little more than a toy, and while the Qpid bridge was an improvement, there has never been a production-quality, open-source transport bridge for OpenMAMA. Until now. NYFIX, a division of Itiviti AB, has recently released an open-source transport for OpenMAMA based on ZeroMQ which we are calling OZ. OZ has been in live production use supporting the NYFIX Marketplace since early March in our data centers in Europe, Asia and the U.S., processing roughly 50 million messages per day. OZ was designed to support many of the most popular features of typical MOM's:
We recently published a whitepaper describing OZ, but the more technical-minded will probably prefer to check out the docs and/or take a look at the sample code. (In particular, the examples use modern C++ constructs to provide a kinder, gentler introduction to OpenMAMA). We're hopeful that the OpenMAMA community will find OZ helpful, and we look forward to working with everyone in the community to make OZ, and OpenMAMA itself, even better. We encourage everyone to check out OZ at https://github.com/nyfix/OZ. Please contact me directly or raise an issue with any comments, suggestions, etc.
|
|
|
|
OpenMAMA 6.3.1 Released
Frank Quinn
Hi Folks,
We are pleased to announce OpenMAMA 6.3.1 is finally here and has now propagated to Cloudsmith, Maven central etc!
This is a maintenance release which fixes several outstanding bugs and introduces some new functionality.
For a complete list of all 21 issues included in this release, please see here: https://github.com/OpenMAMA/OpenMAMA/milestone/11?closed=1
Cheers, Frank
Frank Quinn Cascadium T: +44 (0) 28 8678 8015 E: fquinn@...
|
|
|
|
New OpenMAMA Website is Live!
Frank Quinn
Hi Folks,
After a successful testing period (thanks to all who were involved), we are delighted to announce the official launch the new OpenMAMA website which went live yesterday evening: https://openmama.org
We will be getting the word out on social media so please like / share our posts to extend our reach! Also please update any existing links etc. to the OpenMAMA website where appropriate.
The site is a complete overhaul and has several key improvements over the former site beyond aesthetics:
If there are any issues please raise them at https://github.com/OpenMAMA/openmama.github.io/issues and we hope you enjoy the new site! As usual, we’ll be keeping an eye on gitter if you have any immediate feedback / concerns.
Cheers, Frank
Frank Quinn Cascadium T: +44 (0) 28 8678 8015 E: fquinn@...
|
|
|
|
Re: Last call for OpenMAMA 6.3.1 release candidate
Frank Quinn
Hi Folks,
Looks like a number of issues are still coming in and I have identified a few things that I want to get in myself, so I’m going to push this back 1 week to the weekend commencing 28-Aug 2020.
Cheers, Frank Frank Quinn, Cascadium | +44 (0) 28 8678 8015 | http://cascadium.io
From: Frank Quinn
Sent: 18 August 2020 23:12 To: openmama-users@... Subject: Last call for OpenMAMA 6.3.1 release candidate
Hi Folks,
I plan on taking a cut for OpenMAMA 6.3.1 RC1 over the coming weekend (21-Aug 2020). If anyone has anything they want to submit before then please let me know so I can delay the cut if necessary, otherwise I will continue as planned.
This release includes some changes to support:
It also effectively drops support for the now EOL:
This is in preparation for a RC that will be cut within the next few weeks which is now set up to include RPMs and Debian packages for these modern distros to make installation easy, take care of dependencies etc.
Cheers, Frank
Frank Quinn Cascadium T: +44 (0) 28 8678 8015 E: fquinn@...
|
|
|
|
Last call for OpenMAMA 6.3.1 release candidate
Frank Quinn
Hi Folks,
I plan on taking a cut for OpenMAMA 6.3.1 RC1 over the coming weekend (21-Aug 2020). If anyone has anything they want to submit before then please let me know so I can delay the cut if necessary, otherwise I will continue as planned.
This release includes some changes to support:
It also effectively drops support for the now EOL:
This is in preparation for a RC that will be cut within the next few weeks which is now set up to include RPMs and Debian packages for these modern distros to make installation easy, take care of dependencies etc.
Cheers, Frank
Frank Quinn Cascadium T: +44 (0) 28 8678 8015 E: fquinn@...
|
|
|
|
Use of multiple bridges
Sergey Emantayev
I'm exploring an option for my MAMDA application to receive market data from multiple bridges. However, the Mamda initialisation API like MamdaCommonFields::setDictionary is confusing me because it is the static function accepting a single dictionary. Different bridges may have different dictionaries and it seems to work with only a single bridge. Is there any solution or workaround? Best Regards, Sergey Emantayev
|
|
|
|
OpenMAMA 6.2.3 Released
Frank Quinn <fquinn@...>
Hi Folks,
We are pleased to announce the final release of OpenMAMA 6.2.3 is now available:
https://github.com/OpenMAMA/OpenMAMA/releases/tag/OpenMAMA-6.2.3-release
This is a hotfix release to address two key issues which were discovered as part of the recent 6.2.2 release:
* Restore mamaSubscription RecoverGaps functions accidentally removed in the last release * Restore missing wombat portability headers in 6.2.2 Release
For a complete list of the issues included in this release, please see here: https://github.com/OpenMAMA/OpenMAMA/milestone/9?closed=1
Cheers, Frank
Frank Quinn Cascadium T: +44 (0) 28 8678 8015 E: fquinn@...
|
|
|
|
Re: OpenMAMA 6.2.2 Released
Damian Maguire
Thanks for this Frank, some fantastic work gone into this one. Great to have the release available in so many of the standard repos now, and with Cmake support getting bedded in it should be a lot easier for people to
build from scratch as well (big thanks to Victor for all the effort on that one).
Huge amount of other work in there as well, so thanks to all the contributors along the way.
Thanks,
Damian DAMIAN MAGUIRE
Senior Sales Engineer
Vela
O. +44 289 568 0298 M. +44 783 584 4770 dmaguire@...
Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD tradevela.com | @tradevela
From: Openmama-dev@... <Openmama-dev@...> on behalf of Frank Quinn <fquinn@...>
Sent: 02 July 2018 21:11 To: openmama-dev@...; openmama-users@... Subject: [Openmama-dev] OpenMAMA 6.2.2 Released Hi Folks,
We are pleased to announce the final release of OpenMAMA 6.2.2 is now available:
https://github.com/OpenMAMA/OpenMAMA/releases/tag/OpenMAMA-6.2.2-release
Note that for the first time, a OpenMAMA generally available release is now available via Maven Central, Microsoft’s vcpkg and yum repositories (via Cloudsmith).
Documentation will be coming in the following weeks with more details including how to use our new experimental cmake build system!
This is a maintenance release which fixes several outstanding bugs and introduces some new functionality.
Key features include:
For a complete list of all 55 issues included in this release, please see here: https://github.com/OpenMAMA/OpenMAMA/milestone/8?closed=1
A special thanks to all developers, contributors and testers who helped is getting this out door.
Cheers, Frank
Frank Quinn Cascadium T: +44 (0) 28 8678 8015 E: fquinn@...
The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Systems LLC
|
|
|
|
OpenMAMA 6.2.2 Released
Frank Quinn <fquinn@...>
Hi Folks,
We are pleased to announce the final release of OpenMAMA 6.2.2 is now available:
https://github.com/OpenMAMA/OpenMAMA/releases/tag/OpenMAMA-6.2.2-release
Note that for the first time, a OpenMAMA generally available release is now available via Maven Central, Microsoft’s vcpkg and yum repositories (via Cloudsmith).
Documentation will be coming in the following weeks with more details including how to use our new experimental cmake build system!
This is a maintenance release which fixes several outstanding bugs and introduces some new functionality.
Key features include:
For a complete list of all 55 issues included in this release, please see here: https://github.com/OpenMAMA/OpenMAMA/milestone/8?closed=1
A special thanks to all developers, contributors and testers who helped is getting this out door.
Cheers, Frank
Frank Quinn Cascadium T: +44 (0) 28 8678 8015 E: fquinn@...
|
|
|
|
OpenMAMA-6.2.2-rc2 Now Available
Frank Quinn <fquinn@...>
Hi Folks,
We were planning on releasing today but a pull request landed yesterday containing some bugfixes for the recently added plugin code which I have deemed as necessary for this release, so I have now cut OpenMAMA 6.2.2-rc2 which can be found here:
https://github.com/OpenMAMA/OpenMAMA/releases/tag/OpenMAMA-6.2.2-rc2
Considering this is bugfix only, the release candidate window will be one week, making the target release date (all being well) 28th June.
I encourage all application and bridge developers to rigorously test this new release with their software especially with respect to the market data subscription life cycle.
If there are any further incoming changes please advise me asap to see if the above dates need to be revised.
Cheers, Frank
Frank Quinn, Cascadium | +44 (0) 28 8678 8015 | http://cascadium.io
From: Frank Quinn
Sent: 13 June 2018 20:36 To: 'openmama-dev@...' <openmama-dev@...>; 'openmama-users@...' <openmama-users@...> Subject: RE: OpenMAMA-6.2.2-rc1 Now Available
Hi Folks,
Hope testing is going well!
We’ve just submitted a change to correct a few unit test compiler warnings and memory leaks recently introduced but nothing which impacts core code so there’s no current reason to extend the RC window.
Just a reminder that we’re going into the final week of testing here so if anyone has spotted anything unusual please speak up now if you want a fix to make it into this release!
Cheers, Frank
Frank Quinn, Cascadium | +44 (0) 28 8678 8015 | http://cascadium.io
From: Frank Quinn
Hi Folks,
We are pleased to announce the first release candidate for OpenMAMA 6.2.2 is now available:
https://github.com/OpenMAMA/OpenMAMA/releases/tag/OpenMAMA-6.2.2-rc1
This is a maintenance release which fixes several outstanding bugs and introduces some new functionality.
Key features include:
For a complete list of all 54 issues included in this release, please see here: https://github.com/OpenMAMA/OpenMAMA/milestone/8?closed=1
Thank you all in advance for your help in testing - if you spot any issues, please follow our guidelines for raising an issue, or even better, follow our guidelines for raising a patch.
Since the release is significant, the testing period will be 3 weeks from today making the target release date 20th June and everyone is invited to try it out - binary releases are available at the link above.
If critical issues are found and not resolved before this date, we will continue to go through weekly release candidates until have a stable release ready.
Cheers, Frank
Frank Quinn Cascadium T: +44 (0) 28 8678 8015 E: fquinn@...
|
|
|
|
Re: OpenMAMA-6.2.2-rc1 Now Available
Frank Quinn <fquinn@...>
Hi Folks,
Hope testing is going well!
We’ve just submitted a change to correct a few unit test compiler warnings and memory leaks recently introduced but nothing which impacts core code so there’s no current reason to extend the RC window.
Just a reminder that we’re going into the final week of testing here so if anyone has spotted anything unusual please speak up now if you want a fix to make it into this release!
Cheers, Frank
Frank Quinn, Cascadium | +44 (0) 28 8678 8015 | http://cascadium.io
From: Frank Quinn
Sent: 29 May 2018 21:07 To: 'openmama-dev@...' <openmama-dev@...>; 'openmama-users@...' <openmama-users@...> Subject: OpenMAMA-6.2.2-rc1 Now Available
Hi Folks,
We are pleased to announce the first release candidate for OpenMAMA 6.2.2 is now available:
https://github.com/OpenMAMA/OpenMAMA/releases/tag/OpenMAMA-6.2.2-rc1
This is a maintenance release which fixes several outstanding bugs and introduces some new functionality.
Key features include:
For a complete list of all 54 issues included in this release, please see here: https://github.com/OpenMAMA/OpenMAMA/milestone/8?closed=1
Thank you all in advance for your help in testing - if you spot any issues, please follow our guidelines for raising an issue, or even better, follow our guidelines for raising a patch.
Since the release is significant, the testing period will be 3 weeks from today making the target release date 20th June and everyone is invited to try it out - binary releases are available at the link above.
If critical issues are found and not resolved before this date, we will continue to go through weekly release candidates until have a stable release ready.
Cheers, Frank
Frank Quinn Cascadium T: +44 (0) 28 8678 8015 E: fquinn@...
|
|
|
|
OpenMAMA-6.2.2-rc1 Now Available
Frank Quinn <fquinn@...>
Hi Folks,
We are pleased to announce the first release candidate for OpenMAMA 6.2.2 is now available:
https://github.com/OpenMAMA/OpenMAMA/releases/tag/OpenMAMA-6.2.2-rc1
This is a maintenance release which fixes several outstanding bugs and introduces some new functionality.
Key features include:
For a complete list of all 54 issues included in this release, please see here: https://github.com/OpenMAMA/OpenMAMA/milestone/8?closed=1
Thank you all in advance for your help in testing - if you spot any issues, please follow our guidelines for raising an issue, or even better, follow our guidelines for raising a patch.
Since the release is significant, the testing period will be 3 weeks from today making the target release date 20th June and everyone is invited to try it out - binary releases are available at the link above.
If critical issues are found and not resolved before this date, we will continue to go through weekly release candidates until have a stable release ready.
Cheers, Frank
Frank Quinn Cascadium T: +44 (0) 28 8678 8015 E: fquinn@...
|
|
|
|
Re: [Openmama-dev] Let's test cmake support
Frank Quinn <fquinn@...>
Hi folks,
Further to this, we have now wired up with C#, install rules and unit tests on Linux, Windows and even native OSX (with some recent changes).
This is a huge step because it effectively outsources compiler support which we effectively had to manage ourselves with our previous scons infrastructure as well as fight with python environments.
With that in mind I have now raised https://github.com/OpenMAMA/OpenMAMA/pull/361 which will hopefully make it into next soon!
After the next release goes out (which I propose is soon), we can look at making cmake the default for CI and the release following that one.
Cheers,
Frank
On 27 Apr 2018 10:05, Victor Maleyev <imnotmindlin@...> wrote:
Hi guys,
Me and Frank made some efforts to support CMake build system: it builds MAMA on Linux and Windows. Unfortunately it is not in trunk yet but I desperately need any feedback on how it works to make it stable and ready for release. Just clone the repo from here: https://github.com/fquinner/OpenMAMA/tree/feature-cmake-support and try build it like this: mkdir build cmake .. make Make sure that flex, Apache portable runtime and gradle are installed. Feel free to mail me if issues are found.
|
|
|
|
[Openmama-dev] Let's test cmake support
Victor Maleyev
Added openmama-users -------- Пересылаемое сообщение-------- 27.04.2018, 12:05, "Victor Maleyev" <imnotmindlin@...>: Hi guys, -------- Завершение пересылаемого сообщения --------
|
|
|
|
Re: [Openmama-dev] Did MAMAIgnoreDeprecatedOpen ever work on Linux? [I]
Frank Quinn <fquinn.ni@...>
Hi Yury, Yes that method is deprecated because OpenMAMA prefers dynamic loading methods now (so that payloads don't need to be registered in OpenMAMA's core code to load or depend on magic characters). Have you looked at mamaMsg_createForPayloadBridge? It should do the same thing, but accepts a payload bridge rather than a char identifier and is not deprecated. Damian also provided some helper functions included 6.2.1 to help look up bridges by name which might be helpful for lookup and cache, including mama_getPayloadBridge. Cheers, Frank
On Tue, Oct 3, 2017 at 10:07 AM, Yury Batrakov <yury.batrakov@...> wrote:
|
|
|
|
Re: [Openmama-dev] Questions about MamaStartCallback
Frank Quinn <fquinn.ni@...>
Hi Yury, I did have this proposed change merged, but have since discovered that it causes a deadlock in our unit test on windows. On closer inspection, it looks like the problem is caused by the fact that making mama_stop blocking will effectively deadlock any application which attempts to call mama_stop from a MAMA event coming off the default queue (like our unit test application). Since I expect this is a common enough pattern, we'll have to revert this change. Note that the argument to startBackground is misleadingly a stop callback - i.e. it will be called when mama_start *unblocks*. With this in mind, you should be able to get your MyMamaConnection destructor to wait on a semaphore raised by onStartComplete method (use mama_startBackgroundEx rather than mama_startBackground to provide a closure) to raise a semaphore to simulate the equivalent behaviour and prevent the crash. Cheers, Frank
On Mon, Aug 14, 2017 at 1:21 PM, Yury Batrakov <yury.batrakov@...> wrote: Classification: Public
|
|
|
|
Re: [Openmama-dev] Concurrent subscription.destroy() ? Crash when using tick42rmds transport.
Konstantin Baydarov
Classification: Public
toggle quoted messageShow quoted text
Hi, Tom. As Frank suggested, I switched our app from destroy () to destroyEx() (MamaSubscription), but I still get a segmentation fault. I think the reason of segfault is that while RMDSBridgeSubscription::OnMessage() calls mamaSubscription_processMsg() in the tick42rmds thread - the subscription is destroyed in the mamaQueue_dispatch() by the mama_start() thread. The problem is that tick42rmds processing thread, calling RMDSBridgeSubscription::OnMessage(), is not synchronized with mama_start() thread, so mama_start() thread can destroy mama subscription at any time including in the moment of running mamaSubscription_processMsg(). Here are stacks of both threads that operates on the same subscription concurrently without any synchronization: 1) mama_start() thread that destroys subscription: #0 mamaSubscription_destroy (subscription=0x0) at mama/c_cpp/src/c/subscription.c:2929 #1 0x00007ffff51f259a in tick42rmdsBridgeMamaInbox_destroy () from /export/home/dnauat/komodo/UAT/UK/kb/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #2 0x00007ffff78b836b in mamaInbox_destroy (inbox=0x1065f10) at mama/c_cpp/src/c/inbox.c:249 #3 0x00007ffff78a2aec in imageRequest_destroy (request=0x104a300) at mama/c_cpp/src/c/imagerequest.c:425 #4 0x00007ffff78a1425 in dqContext_cleanup (ctx=0x7fffec3add10) at mama/c_cpp/src/c/dqstrategy.c:621 #5 0x00007ffff78cc4d3 in mamaSubscription_cleanup (subscription=0x7fffec3adae0) at mama/c_cpp/src/c/subscription.c:1510 #6 0x00007ffff78ce5df in mamaSubscription_destroy (subscription=0x7fffec3adae0) at mama/c_cpp/src/c/subscription.c:2948 #7 0x00007ffff78cc7b6 in mamaSubscription_DestroyThroughQueueCB (Queue=0x61e340, closure=0x7fffec3adae0) at mama/c_cpp/src/c/subscription.c:1604 #8 0x00007ffff78e634f in wombatQueue_dispatchInt (queue=0x61e430, data=0x0, closure=0x0, isTimed=1 '\001', timout=500) at common/c_cpp/src/c/queue.c:319 #9 0x00007ffff78e63d2 in wombatQueue_timedDispatch (queue=0x61e430, data=0x0, closure=0x0, timeout=500) at common/c_cpp/src/c/queue.c:335 #10 0x00007ffff51fa6d9 in tick42rmdsBridgeMamaQueue_dispatch () from /export/home/dnauat/komodo/UAT/UK/kb/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #11 0x00007ffff78c3a8b in mamaQueue_dispatch (queue=0x61e340) at mama/c_cpp/src/c/queue.c:819 #12 0x00007ffff51f20a8 in tick42rmdsBridge_start () from /export/home/dnauat/komodo/UAT/UK/kb/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #13 0x00007ffff78a8810 in mama_start (bridgeImpl=0x61d660) at mama/c_cpp/src/c/mama.c:1591 #14 0x00007ffff7b8ee53 in Wombat::Mama::start (bridgeImpl=0x61d660) at mama/c_cpp/src/cpp/mamacpp.cpp:198 #15 0x000000000040c53a in MamaListen::start (this=0x7fffffffd690) at mamalistencpp_mt.cpp:1072 #16 0x000000000040ebaf in main (argc=9, argv=0x7fffffffd948) at mamalistencpp_mt.cpp:1915 2) tick42rmds thread - moment of Segfault: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff38ff700 (LWP 25476)] 0x00007ffff78a3508 in imageRequest_stopWaitForResponse (request=0x0) at mama/c_cpp/src/c/imagerequest.c:772 772 mamaSubscription_getTransport ( impl->mSubscription, &tport ); (gdb) bt #0 0x00007ffff78a3508 in imageRequest_stopWaitForResponse (request=0x0) at mama/c_cpp/src/c/imagerequest.c:772 #1 0x00007ffff78cbe53 in mamaSubscription_stopWaitForResponse (subscription=0x7fffe0587200, ctx=0x7fffe0587420) at mama/c_cpp/src/c/subscription.c:1264 #2 0x00007ffff78a393e in processPointToPointMessage (callback=0x7fffe0587a00, msg=0x63eda0, msgType=6, ctx=0x7fffe0587420) at mama/c_cpp/src/c/listenermsgcallback.c:171 #3 0x00007ffff78a3fdc in listenerMsgCallback_processMsg (callback=0x7fffe0587a00, msg=0x63eda0, ctx=0x7fffe0587420) at mama/c_cpp/src/c/listenermsgcallback.c:482 #4 0x00007ffff78cd9a7 in _mamaSubscription_processMsg (subscription=0x7fffe0587200, msg=0x63eda0) at mama/c_cpp/src/c/subscription.c:2322 #5 0x00007ffff78cd7ff in mamaSubscription_processMsg (subscription=0x7fffe0587200, msg=0x63eda0) at mama/c_cpp/src/c/subscription.c:2272 #6 0x00007ffff51fae33 in RMDSBridgeSubscription::OnMessage(mamaMsgImpl_*, mamaMsgType) () from /export/home/dnauat/komodo/UAT/UK/kb/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #7 0x00007ffff5257632 in UPASubscription::NotifyListenersRefreshMessage(mamaMsgImpl_*, boost::shared_ptr<RMDSBridgeSubscription>, bool) () from /export/home/dnauat/komodo/UAT/UK/kb/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #8 0x00007ffff525bad2 in UPASubscription::InternalProcessMarketPriceResponse(RsslMsg*, RsslDecIterator*) () from /export/home/dnauat/komodo/UAT/UK/kb/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #9 0x00007ffff52611d5 in UPASubscription::ProcessMarketPriceResponse(RsslMsg*, RsslDecIterator*) () from /export/home/dnauat/komodo/UAT/UK/kb/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #10 0x00007ffff522a268 in UPAConsumer::ProcessResponse(RsslChannel*, RwfBuffer*) () from /export/home/dnauat/komodo/UAT/UK/kb/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #11 0x00007ffff522a850 in UPAConsumer::ReadFromChannel(RsslChannel*) () from /export/home/dnauat/komodo/UAT/UK/kb/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #12 0x00007ffff522bb1d in UPAConsumer::Run() () from /export/home/dnauat/komodo/UAT/UK/kb/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #13 0x00007ffff5211b1c in RMDSSubscriber::threadFunc(void*) () from /export/home/dnauat/komodo/UAT/UK/kb/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #14 0x00007ffff6733806 in start_thread () from /lib64/libpthread.so.0 #15 0x00007ffff5af59bd in clone () from /lib64/libc.so.6 #16 0x0000000000000000 in ?? () BR, Konstantin Baydarov
-----Original Message-----
From: Tom Doust [mailto:tom.doust@tick42.com] Sent: Tuesday, June 20, 2017 8:40 PM To: Konstantin Baydarov <konstantin.baydarov@db.com>; Yury Batrakov <yury.batrakov@db.com>; openmama-dev <openmama-dev@lists.openmama.org>; openmama-users <openmama-users@lists.openmama.org> Subject: Re: [Openmama-dev] [Openmama-users] Concurrent subscription.destroy() ? Crash when using tick42rmds transport. Yes we call back on the bridge’s thread. We don’t queue the message, we leave the client code to do that if it wants to. This is by design. On 20/06/2017, 18:18, "Konstantin Baydarov" <konstantin.baydarov@db.com> wrote: Classification: Public Hi, Tom. I noticed, that comparing to qpid bridge(that comes with openmama sources), tick42rmds calls mamaSubscription_processMsg() method from separate thread and not from mamaQueue_dispatch(), wondering if it's correct. Probably it's one of the reasons of the issue that we facing? Qpid bridge call stack: #0 mamaSubscription_processMsg (subscription=0x76e150, msg=0x7aebb0) at mama/c_cpp/src/c/subscription.c:2226 #1 0x00007ffff7b4c580 in imageRequestImpl_onInitialMessage (msg=0x7aebb0, closure=0x772710) at mama/c_cpp/src/c/imagerequest.c:225 #2 0x00007ffff648bede in qpidBridgeMamaInboxImpl_onMsg (subscription=0x772900, msg=0x7aebb0, closure=0x7727c0, itemClosure=0x0) at mama/c_cpp/src/c/bridge/qpid/inbox.c:298 #3 0x00007ffff7b76ab4 in mamaSubscription_forwardMsg (subscription=0x772900, msg=0x7aebb0) at mama/c_cpp/src/c/subscription.c:1422 #4 0x00007ffff7b781ef in mamaSubscription_processMsg (subscription=0x772900, msg=0x7aebb0) at mama/c_cpp/src/c/subscription.c:2315 #5 0x00007ffff6490818 in qpidBridgeMamaTransportImpl_queueCallback (queue=0x60eb50, closure=0x61df80) at mama/c_cpp/src/c/bridge/qpid/transport.c:1083 #6 0x00007ffff7b90a1f in wombatQueue_dispatchInt (queue=0x60ecb0, data=0x0, closure=0x0, isTimed=1 '\001', timout=500) at common/c_cpp/src/c/queue.c:319 #7 0x00007ffff7b90aa2 in wombatQueue_timedDispatch (queue=0x60ecb0, data=0x0, closure=0x0, timeout=500) at common/c_cpp/src/c/queue.c:335 #8 0x00007ffff648e720 in qpidBridgeMamaQueue_dispatch (queue=0x60ec40) at mama/c_cpp/src/c/bridge/qpid/queue.c:265 #9 0x00007ffff7b6e1de in mamaQueue_dispatch (queue=0x60eb50) at mama/c_cpp/src/c/queue.c:824 #10 0x00007ffff648a8c3 in qpidBridge_start (defaultEventQueue=0x60eb50) at mama/c_cpp/src/c/bridge/qpid/bridge.c:196 #11 0x00007ffff7b52976 in mama_start (bridgeImpl=0x60e750) at mama/c_cpp/src/c/mama.c:1659 #12 0x0000000000403e61 in buildDataDictionary () at mama/c_cpp/src/examples/c/mamalistenc.c:647 #13 0x000000000040366f in main (argc=9, argv=0x7fffffffd728) at mama/c_cpp/src/examples/c/mamalistenc.c:335 tick42rmds bridge call stack: #0 0x0000000000000000 in ?? () #1 0x00007ffff78cc259 in mamaSubscription_forwardMsg (subscription=0x7fffe9757c50, msg=0x641d60) at mama/c_cpp/src/c/subscription.c:1426 #2 0x00007ffff78a38ec in processPointToPointMessage (callback=0x7fffe9768fb0, msg=0x641d60, msgType=6, ctx=0x7fffe939b500) at mama/c_cpp/src/c/listenermsgcallback.c:174 #3 0x00007ffff78a3f37 in listenerMsgCallback_processMsg (callback=0x7fffe9768fb0, msg=0x641d60, ctx=0x7fffe939b500) at mama/c_cpp/src/c/listenermsgcallback.c:471 #4 0x00007ffff78cd7e5 in mamaSubscription_processMsg (subscription=0x7fffe939b2e0, msg=0x641d60) at mama/c_cpp/src/c/subscription.c:2260 #5 0x00007ffff51fdaf3 in RMDSBridgeSubscription::OnMessage(mamaMsgImpl_*, mamaMsgType) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #6 0x00007ffff5258492 in UPASubscription::NotifyListenersRefreshMessage(mamaMsgImpl_*, boost::shared_ptr<RMDSBridgeSubscription>, bool) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #7 0x00007ffff5259032 in UPASubscription::InternalProcessMarketPriceResponse(RsslMsg*, RsslDecIterator*) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #8 0x00007ffff525e785 in UPASubscription::ProcessMarketPriceResponse(RsslMsg*, RsslDecIterator*) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #9 0x00007ffff522ccc8 in UPAConsumer::ProcessResponse(RsslChannel*, RwfBuffer*) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #10 0x00007ffff522d2b0 in UPAConsumer::ReadFromChannel(RsslChannel*) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #11 0x00007ffff522e62d in UPAConsumer::Run() () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #12 0x00007ffff52146cc in RMDSSubscriber::threadFunc(void*) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #13 0x00007ffff6733806 in start_thread () from /lib64/libpthread.so.0 #14 0x00007ffff5af59bd in clone () from /lib64/libc.so.6 #15 0x0000000000000000 in ?? () BR, Konstantin Baydarov -----Original Message----- From: Yury Batrakov Sent: Tuesday, June 20, 2017 8:12 PM To: Tom Doust <tom.doust@tick42.com>; Konstantin Baydarov <konstantin.baydarov@db.com>; openmama-dev <openmama-dev@lists.openmama.org>; openmama-users <openmama-users@lists.openmama.org> Subject: RE: [Openmama-dev] [Openmama-users] Concurrent subscription.destroy() ? Crash when using tick42rmds transport. Classification: Public Hi Tom, We are using version 1.3. As I see from latest github code the problem still exists. See RMDSBridgeSubscription::OnMessage method: if (isShutdown_ || ((0 != source_) && source_->IsPausedUpdates())) { return; } // ... Shutdown() may be called here // And then MAMA can start destroying subscription_'s fields try { status = mamaSubscription_processMsg(subscription_, msg); // This function will be examining subscription_'s fields being destroyed } -----Original Message----- From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Tom Doust Sent: Tuesday, June 20, 2017 4:09 PM To: Konstantin Baydarov <konstantin.baydarov@db.com>; openmama-dev <openmama-dev@lists.openmama.org>; openmama-users <openmama-users@lists.openmama.org> Subject: Re: [Openmama-dev] [Openmama-users] Concurrent subscription.destroy() ? Crash when using tick42rmds transport. Hi Yury, Konstantin Are you using the current github version of the bridge code? We looked at and fixed some of the issues around locking the subscription destroy some time back. It would be good to know if we have missed something. Best regards Tom -----Original Message----- From: openmama-users-bounces@lists.openmama.org [mailto:openmama-users-bounces@lists.openmama.org] On Behalf Of Konstantin Baydarov Sent: 20 June 2017 11:32 To: openmama-dev <openmama-dev@lists.openmama.org>; openmama-users <openmama-users@lists.openmama.org> Subject: Re: [Openmama-users] Concurrent subscription.destroy() ? Crash when using tick42rmds transport. Classification: Public Hi, guys. I'm working on the issue with Yury. I spotted the deadlock possibility during debugging tick42rmds bridge crash on unsubscribe, will be interested knowing the solution as well. BR, Konstantin Baydarov -----Original Message----- From: Yury Batrakov Sent: Tuesday, June 20, 2017 12:38 PM To: Frank Quinn <fquinn@velatt.com>; openmama-dev <openmama-dev@lists.openmama.org>; openmama-users <openmama-users@lists.openmama.org> Cc: Konstantin Baydarov <konstantin.baydarov@db.com> Subject: RE: Concurrent subscription.destroy() ? Crash when using tick42rmds transport. Classification: Public Hi Frank, Let me answer your question in random order :) 2. It looks like a designed behavior of RMDS bridge - callbacks are invoked in a thread servicing transport events from a server. One thread per mamaTransport is created. 1. Therefore race conditions are possible. Our case is: mute is called for a subscription, then mamaSubscription_cleanup frees self->mInitialRequest but concurrent mamaSubscription_processMsg call tries to access self->mInitialRequest because the message it processes is initial message from a server. 3. Taking in account p.2 we cannot process mute request asynchronously as MAMA starts freeing subscription resources immediately after mute is called. Do you think it is possible for MAMA not to invoke bridgeMamaSubscriptionMute() under mamaSubscription locks? Thus the contract for this function would be: - This call should be synchronous and no events should be processed after it returned (like before) - It should be reentrant and synchronize it's operations by itself (quite sensible requirement) -----Original Message----- From: Frank Quinn [mailto:fquinn@velatt.com] Sent: Tuesday, June 20, 2017 10:05 AM To: Yury Batrakov <yury.batrakov@db.com>; openmama-dev <openmama-dev@lists.openmama.org>; openmama-users <openmama-users@lists.openmama.org> Subject: RE: Concurrent subscription.destroy() ? Crash when using tick42rmds transport. Hi Yury, Thanks for the detailed query, I have a few outstanding questions and suggestions on this one: 1. I would question whether or not mamaSubscription_processMsg should crash for a muted subscription. Muting is a state which exists to attempt to stop new events coming in. However if you're in the dispatcher thread and you just received an object, it's too late this time - mute should only be invoked prior to prevent the *next* read. So perhaps (knowing nothing about the RMDS bridge) the straightforward solution would be simply to do the checks which may cause muting after the callback? 2. Should the thread which processes events from RMDS server invoke the callback method directly (inline)? Is it on the same thread as is assigned to the MAMA Subscription object? It should be to match the application's expected concurrency behaviour. 3. Rather than muting immediately, you could consider creating a muting callback event which gets enqueued onto your subscription thread. That way the mute event will always be synchronous with the subscription thread and you don't need to worry about locking any associated resources. Locking with subscription objects (particularly MAMA core objects) is hairy stuff and you should avoid it where possible to avoid race conditions. Cheers, Frank -----Original Message----- From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury Batrakov Sent: 19 June 2017 17:42 To: openmama-dev <openmama-dev@lists.openmama.org>; openmama-users <openmama-users@lists.openmama.org> Subject: [Openmama-dev] Concurrent subscription.destroy() ? Crash when using tick42rmds transport. Classification: Public Hi guys, I've faced the following issue when using OpenMAMA and tick42rmds bridge. The bridge internally creates a thread to process events from RMDS server, once a message is received that thread invokes mamaSubscription_processMsg. While the message is processed user may want to destroy the subscription (obviously in other thread). To avoid corruption of mamaSubscription object, mamaSubscription_destroy() function calls bridge->mute for the bridge to stop calling mamaSubscription_processMsg() and only then deallocates mamaSubscription. The problem with this approach is the following: here's the pseudo code for RMDS dispatching thread if(muted) { // Do not dispatch return; } // Do some other checks <-- mute() may be invoked here mamaSubscription_processMsg() // processMsg for muted subscription, may crash The solution for that is to change RMDS bridge to block in bridge->mute() call until mamaSubscription_processMsg() returns but there's another problem: mamaSubscription_processMsg and mamaSubscription_deactivate may deadlock on wombatThrottle. Consider the following scenario: 1. RMDS bridge thread invokes mamaSubscription_processMsg() for message of type initial 2. User thread invokes mamaSubscription_destroy() which acquires wombat throttle lock: if (impl->mTransport) throttle = mamaTransportImpl_getThrottle(impl->mTransport, MAMA_THROTTLE_DEFAULT); if(NULL != throttle) { wombatThrottle_lock(throttle); } 3. Then mamaSubscription_destroy calls mamaSubscription_deactivate_internal which calls our new version of bridge->mute() which waits for RMDS bridge thread to finish message processing if (impl->mSubscBridge) { impl->mBridgeImpl->bridgeMamaSubscriptionMute (impl->mSubscBridge); } 4. RMDS bridge handles initial message and tries to acquire the same throttle: #5 0x00007ffff78d4f32 in wombatThrottle_lock (throttle=0x6298e0) at mama/c_cpp/src/c/throttle.c:441 #6 0x00007ffff78a34e2 in imageRequest_stopWaitForResponse (request=0x14d1a20) at mama/c_cpp/src/c/imagerequest.c:774 #7 0x00007ffff78cbe06 in mamaSubscription_stopWaitForResponse (subscription=0xe36280, ctx=0xe364a0) at mama/c_cpp/src/c/subscription.c:1262 #8 0x00007ffff78a38fe in processPointToPointMessage (callback=0x1527e50, msg=0x642460, msgType=6, ctx=0xe364a0) at mama/c_cpp/src/c/listenermsgcallback.c:169 #9 0x00007ffff78a3f9c in listenerMsgCallback_processMsg (callback=0x1527e50, msg=0x642460, ctx=0xe364a0) at mama/c_cpp/src/c/listenermsgcallback.c:480 #10 0x00007ffff78cd825 in mamaSubscription_processMsg (subscription=0xe36280, msg=0x642460) at mama/c_cpp/src/c/subscription.c:2259 What do you think is the best way to avoid this? --- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Please refer to https://www.db.com/disclosures for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy. The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC --- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Please refer to https://www.db.com/disclosures for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy. _______________________________________________ Openmama-users mailing list Openmama-users@lists.openmama.org https://lists.openmama.org/mailman/listinfo/openmama-users _______________________________________________ Openmama-dev mailing list Openmama-dev@lists.openmama.org https://lists.openmama.org/mailman/listinfo/openmama-dev --- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Please refer to https://www.db.com/disclosures for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy. --- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Please refer to https://www.db.com/disclosures for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.
|
|
|
|
Re: [Openmama-dev] Concurrent subscription.destroy() ? Crash when using tick42rmds transport.
Tom Doust
Yes we call back on the bridge’s thread. We don’t queue the message, we leave the client code to do that if it wants to. This is by design.
toggle quoted messageShow quoted text
On 20/06/2017, 18:18, "Konstantin Baydarov" <konstantin.baydarov@db.com> wrote:
Classification: Public Hi, Tom. I noticed, that comparing to qpid bridge(that comes with openmama sources), tick42rmds calls mamaSubscription_processMsg() method from separate thread and not from mamaQueue_dispatch(), wondering if it's correct. Probably it's one of the reasons of the issue that we facing? Qpid bridge call stack: #0 mamaSubscription_processMsg (subscription=0x76e150, msg=0x7aebb0) at mama/c_cpp/src/c/subscription.c:2226 #1 0x00007ffff7b4c580 in imageRequestImpl_onInitialMessage (msg=0x7aebb0, closure=0x772710) at mama/c_cpp/src/c/imagerequest.c:225 #2 0x00007ffff648bede in qpidBridgeMamaInboxImpl_onMsg (subscription=0x772900, msg=0x7aebb0, closure=0x7727c0, itemClosure=0x0) at mama/c_cpp/src/c/bridge/qpid/inbox.c:298 #3 0x00007ffff7b76ab4 in mamaSubscription_forwardMsg (subscription=0x772900, msg=0x7aebb0) at mama/c_cpp/src/c/subscription.c:1422 #4 0x00007ffff7b781ef in mamaSubscription_processMsg (subscription=0x772900, msg=0x7aebb0) at mama/c_cpp/src/c/subscription.c:2315 #5 0x00007ffff6490818 in qpidBridgeMamaTransportImpl_queueCallback (queue=0x60eb50, closure=0x61df80) at mama/c_cpp/src/c/bridge/qpid/transport.c:1083 #6 0x00007ffff7b90a1f in wombatQueue_dispatchInt (queue=0x60ecb0, data=0x0, closure=0x0, isTimed=1 '\001', timout=500) at common/c_cpp/src/c/queue.c:319 #7 0x00007ffff7b90aa2 in wombatQueue_timedDispatch (queue=0x60ecb0, data=0x0, closure=0x0, timeout=500) at common/c_cpp/src/c/queue.c:335 #8 0x00007ffff648e720 in qpidBridgeMamaQueue_dispatch (queue=0x60ec40) at mama/c_cpp/src/c/bridge/qpid/queue.c:265 #9 0x00007ffff7b6e1de in mamaQueue_dispatch (queue=0x60eb50) at mama/c_cpp/src/c/queue.c:824 #10 0x00007ffff648a8c3 in qpidBridge_start (defaultEventQueue=0x60eb50) at mama/c_cpp/src/c/bridge/qpid/bridge.c:196 #11 0x00007ffff7b52976 in mama_start (bridgeImpl=0x60e750) at mama/c_cpp/src/c/mama.c:1659 #12 0x0000000000403e61 in buildDataDictionary () at mama/c_cpp/src/examples/c/mamalistenc.c:647 #13 0x000000000040366f in main (argc=9, argv=0x7fffffffd728) at mama/c_cpp/src/examples/c/mamalistenc.c:335 tick42rmds bridge call stack: #0 0x0000000000000000 in ?? () #1 0x00007ffff78cc259 in mamaSubscription_forwardMsg (subscription=0x7fffe9757c50, msg=0x641d60) at mama/c_cpp/src/c/subscription.c:1426 #2 0x00007ffff78a38ec in processPointToPointMessage (callback=0x7fffe9768fb0, msg=0x641d60, msgType=6, ctx=0x7fffe939b500) at mama/c_cpp/src/c/listenermsgcallback.c:174 #3 0x00007ffff78a3f37 in listenerMsgCallback_processMsg (callback=0x7fffe9768fb0, msg=0x641d60, ctx=0x7fffe939b500) at mama/c_cpp/src/c/listenermsgcallback.c:471 #4 0x00007ffff78cd7e5 in mamaSubscription_processMsg (subscription=0x7fffe939b2e0, msg=0x641d60) at mama/c_cpp/src/c/subscription.c:2260 #5 0x00007ffff51fdaf3 in RMDSBridgeSubscription::OnMessage(mamaMsgImpl_*, mamaMsgType) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #6 0x00007ffff5258492 in UPASubscription::NotifyListenersRefreshMessage(mamaMsgImpl_*, boost::shared_ptr<RMDSBridgeSubscription>, bool) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #7 0x00007ffff5259032 in UPASubscription::InternalProcessMarketPriceResponse(RsslMsg*, RsslDecIterator*) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #8 0x00007ffff525e785 in UPASubscription::ProcessMarketPriceResponse(RsslMsg*, RsslDecIterator*) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #9 0x00007ffff522ccc8 in UPAConsumer::ProcessResponse(RsslChannel*, RwfBuffer*) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #10 0x00007ffff522d2b0 in UPAConsumer::ReadFromChannel(RsslChannel*) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #11 0x00007ffff522e62d in UPAConsumer::Run() () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #12 0x00007ffff52146cc in RMDSSubscriber::threadFunc(void*) () from /home/gedeapp/baydkon/apps/dbmama/dbmama-api-1.7.1462_dev/lib/libmamatick42rmdsimpl.so #13 0x00007ffff6733806 in start_thread () from /lib64/libpthread.so.0 #14 0x00007ffff5af59bd in clone () from /lib64/libc.so.6 #15 0x0000000000000000 in ?? () BR, Konstantin Baydarov -----Original Message----- From: Yury Batrakov Sent: Tuesday, June 20, 2017 8:12 PM To: Tom Doust <tom.doust@tick42.com>; Konstantin Baydarov <konstantin.baydarov@db.com>; openmama-dev <openmama-dev@lists.openmama.org>; openmama-users <openmama-users@lists.openmama.org> Subject: RE: [Openmama-dev] [Openmama-users] Concurrent subscription.destroy() ? Crash when using tick42rmds transport. Classification: Public Hi Tom, We are using version 1.3. As I see from latest github code the problem still exists. See RMDSBridgeSubscription::OnMessage method: if (isShutdown_ || ((0 != source_) && source_->IsPausedUpdates())) { return; } // ... Shutdown() may be called here // And then MAMA can start destroying subscription_'s fields try { status = mamaSubscription_processMsg(subscription_, msg); // This function will be examining subscription_'s fields being destroyed } -----Original Message----- From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Tom Doust Sent: Tuesday, June 20, 2017 4:09 PM To: Konstantin Baydarov <konstantin.baydarov@db.com>; openmama-dev <openmama-dev@lists.openmama.org>; openmama-users <openmama-users@lists.openmama.org> Subject: Re: [Openmama-dev] [Openmama-users] Concurrent subscription.destroy() ? Crash when using tick42rmds transport. Hi Yury, Konstantin Are you using the current github version of the bridge code? We looked at and fixed some of the issues around locking the subscription destroy some time back. It would be good to know if we have missed something. Best regards Tom -----Original Message----- From: openmama-users-bounces@lists.openmama.org [mailto:openmama-users-bounces@lists.openmama.org] On Behalf Of Konstantin Baydarov Sent: 20 June 2017 11:32 To: openmama-dev <openmama-dev@lists.openmama.org>; openmama-users <openmama-users@lists.openmama.org> Subject: Re: [Openmama-users] Concurrent subscription.destroy() ? Crash when using tick42rmds transport. Classification: Public Hi, guys. I'm working on the issue with Yury. I spotted the deadlock possibility during debugging tick42rmds bridge crash on unsubscribe, will be interested knowing the solution as well. BR, Konstantin Baydarov -----Original Message----- From: Yury Batrakov Sent: Tuesday, June 20, 2017 12:38 PM To: Frank Quinn <fquinn@velatt.com>; openmama-dev <openmama-dev@lists.openmama.org>; openmama-users <openmama-users@lists.openmama.org> Cc: Konstantin Baydarov <konstantin.baydarov@db.com> Subject: RE: Concurrent subscription.destroy() ? Crash when using tick42rmds transport. Classification: Public Hi Frank, Let me answer your question in random order :) 2. It looks like a designed behavior of RMDS bridge - callbacks are invoked in a thread servicing transport events from a server. One thread per mamaTransport is created. 1. Therefore race conditions are possible. Our case is: mute is called for a subscription, then mamaSubscription_cleanup frees self->mInitialRequest but concurrent mamaSubscription_processMsg call tries to access self->mInitialRequest because the message it processes is initial message from a server. 3. Taking in account p.2 we cannot process mute request asynchronously as MAMA starts freeing subscription resources immediately after mute is called. Do you think it is possible for MAMA not to invoke bridgeMamaSubscriptionMute() under mamaSubscription locks? Thus the contract for this function would be: - This call should be synchronous and no events should be processed after it returned (like before) - It should be reentrant and synchronize it's operations by itself (quite sensible requirement) -----Original Message----- From: Frank Quinn [mailto:fquinn@velatt.com] Sent: Tuesday, June 20, 2017 10:05 AM To: Yury Batrakov <yury.batrakov@db.com>; openmama-dev <openmama-dev@lists.openmama.org>; openmama-users <openmama-users@lists.openmama.org> Subject: RE: Concurrent subscription.destroy() ? Crash when using tick42rmds transport. Hi Yury, Thanks for the detailed query, I have a few outstanding questions and suggestions on this one: 1. I would question whether or not mamaSubscription_processMsg should crash for a muted subscription. Muting is a state which exists to attempt to stop new events coming in. However if you're in the dispatcher thread and you just received an object, it's too late this time - mute should only be invoked prior to prevent the *next* read. So perhaps (knowing nothing about the RMDS bridge) the straightforward solution would be simply to do the checks which may cause muting after the callback? 2. Should the thread which processes events from RMDS server invoke the callback method directly (inline)? Is it on the same thread as is assigned to the MAMA Subscription object? It should be to match the application's expected concurrency behaviour. 3. Rather than muting immediately, you could consider creating a muting callback event which gets enqueued onto your subscription thread. That way the mute event will always be synchronous with the subscription thread and you don't need to worry about locking any associated resources. Locking with subscription objects (particularly MAMA core objects) is hairy stuff and you should avoid it where possible to avoid race conditions. Cheers, Frank -----Original Message----- From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury Batrakov Sent: 19 June 2017 17:42 To: openmama-dev <openmama-dev@lists.openmama.org>; openmama-users <openmama-users@lists.openmama.org> Subject: [Openmama-dev] Concurrent subscription.destroy() ? Crash when using tick42rmds transport. Classification: Public Hi guys, I've faced the following issue when using OpenMAMA and tick42rmds bridge. The bridge internally creates a thread to process events from RMDS server, once a message is received that thread invokes mamaSubscription_processMsg. While the message is processed user may want to destroy the subscription (obviously in other thread). To avoid corruption of mamaSubscription object, mamaSubscription_destroy() function calls bridge->mute for the bridge to stop calling mamaSubscription_processMsg() and only then deallocates mamaSubscription. The problem with this approach is the following: here's the pseudo code for RMDS dispatching thread if(muted) { // Do not dispatch return; } // Do some other checks <-- mute() may be invoked here mamaSubscription_processMsg() // processMsg for muted subscription, may crash The solution for that is to change RMDS bridge to block in bridge->mute() call until mamaSubscription_processMsg() returns but there's another problem: mamaSubscription_processMsg and mamaSubscription_deactivate may deadlock on wombatThrottle. Consider the following scenario: 1. RMDS bridge thread invokes mamaSubscription_processMsg() for message of type initial 2. User thread invokes mamaSubscription_destroy() which acquires wombat throttle lock: if (impl->mTransport) throttle = mamaTransportImpl_getThrottle(impl->mTransport, MAMA_THROTTLE_DEFAULT); if(NULL != throttle) { wombatThrottle_lock(throttle); } 3. Then mamaSubscription_destroy calls mamaSubscription_deactivate_internal which calls our new version of bridge->mute() which waits for RMDS bridge thread to finish message processing if (impl->mSubscBridge) { impl->mBridgeImpl->bridgeMamaSubscriptionMute (impl->mSubscBridge); } 4. RMDS bridge handles initial message and tries to acquire the same throttle: #5 0x00007ffff78d4f32 in wombatThrottle_lock (throttle=0x6298e0) at mama/c_cpp/src/c/throttle.c:441 #6 0x00007ffff78a34e2 in imageRequest_stopWaitForResponse (request=0x14d1a20) at mama/c_cpp/src/c/imagerequest.c:774 #7 0x00007ffff78cbe06 in mamaSubscription_stopWaitForResponse (subscription=0xe36280, ctx=0xe364a0) at mama/c_cpp/src/c/subscription.c:1262 #8 0x00007ffff78a38fe in processPointToPointMessage (callback=0x1527e50, msg=0x642460, msgType=6, ctx=0xe364a0) at mama/c_cpp/src/c/listenermsgcallback.c:169 #9 0x00007ffff78a3f9c in listenerMsgCallback_processMsg (callback=0x1527e50, msg=0x642460, ctx=0xe364a0) at mama/c_cpp/src/c/listenermsgcallback.c:480 #10 0x00007ffff78cd825 in mamaSubscription_processMsg (subscription=0xe36280, msg=0x642460) at mama/c_cpp/src/c/subscription.c:2259 What do you think is the best way to avoid this? --- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Please refer to https://www.db.com/disclosures for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy. The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC --- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Please refer to https://www.db.com/disclosures for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy. _______________________________________________ Openmama-users mailing list Openmama-users@lists.openmama.org https://lists.openmama.org/mailman/listinfo/openmama-users _______________________________________________ Openmama-dev mailing list Openmama-dev@lists.openmama.org https://lists.openmama.org/mailman/listinfo/openmama-dev --- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Please refer to https://www.db.com/disclosures for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.
|
|
|