Hi Bill,

Sure sounds good, though these are MAMA bridge level configurations so let's make it:

mama.middleware.<id>.unload_on_close=true|false

and

mama.payload.<id>.unload_on_close=true|false

Similar to https://github.com/OpenMAMA/OpenMAMA/issues/135

The destructor / finalizer approach will fix the GC problem just fine for C# / Java. What is being described there is effectively a bug in those methods. What it won't solve is applications deliberately accessing MAMA payload and middleware methods after close and expecting it to actually function with new messages, iterators etc. However it would certainly stop GC after close from crashing the jvm which sounds like it is Yury's main issue.

As much as I would love to just mandate that all Java and C# developers call destroy on all of the objects (it would make my life a lot easier), a lot of developers we have worked with in the past simply don't accept that as a solution for a GC based runtime which is why we need all this finalizer fun to play nicely.

Cheers,

Frank

On Thu, Dec 28, 2017 at 7:17 PM, Bill Torpey <wallstprog@...> wrote:

Hi Frank:

Why don’t we start over here and see if we can make some progress?

If you’re OK with a solution based on a mama.properties setting, I can live with that.  

For the transport it seems to me that a property setting like the following could work:

mama.<middleware>.transport.<name>.unload_on_close=0|1

And for the payload library:

mama.payload.<id>.unload_on_close=0|1

If that works for you, I’ll work up a PR and maybe we can put this bad boy to bed ;-)

Or feel free to suggest something different.  (With the caveat that the dtor approach is not something I’m open to — nor does it solve the problem for our friends in Java and .Net land).

Regards,

Bill

P.S.  Note also that short-circuiting the dlclose plays much more nicely with tools like valgrind, which discards symbols when a shared library is closed.

On Dec 28, 2017, at 1:27 PM, Frank Quinn <frank@...> wrote:

I have been quite receptive to your concerns - just not your proposed solution which I think is too coarse.

Checking if something exists before trying to access it within a framework is far from a hack - it's simply defensive code. mamaInternal_findPayload is already used in MAMA C++ - this is a solvable problem.

I've clearly acknowledged that it's a problem. I'm not sure what makes you think that I think that it's not. The ticket is still open and i'm suggesting solutions including one which will do exactly what you want based on a configuration option rather than forcing it to be the only option when C developers (for example) would have no need for it whatsoever.

The other is an attempt to solve the problem in a defensive way which means that application developers don't have to give special consideration to this use case when writing their applications.

Cheers,

Frank

On Thu, Dec 28, 2017 at 5:58 PM, Bill Torpey <wallstprog@...> wrote:

Hi Frank:

Obviously I would prefer to remain as close as possible to upstream code, so forking is a last resort.  That being said, you’ve been so far quite unreceptive to my concerns about crashing as a result of unloading the payload (and transport, for that matter) library code.

The approach of checking if the payload library exists is a bit of a hack, and is not possible currently, since the mamaInternal_findPayload function is not exposed as part of the API.  Are you planning to expose those functions?

A better solution I believe would be separating out the unloading of the library code from the other activities needed to do a clean shutdown.  I’ve created a fugly hack for that in the forked OpenMAMA code (see attached), but would much prefer something a bit more “elegant”, which would likely require changing the API, and I don’t want to do that without agreement from you.  But so far you haven’t even agreed that this is a problem.

I’m open to ideas, but I think we can do better than what you suggest.

Regards,

Bill

On Dec 28, 2017, at 12:21 PM, Frank Quinn <frank@...> wrote:

On the contrary I welcome OpenMAMA forks If you want to fork and maintain all fixes etc from the upstream that's entirely your decision and part of the beauty of Open Source software. More power to you.

I simply don't agree that always leaking is the right solution when it's avoidable. It's as simple as that.

After another look I think the solution should live in the C++ destructor method where it can:

1. Cache the payload type in the C++ layer as soon as it is known.

2. On destructor, find out if the bridge referred to in #1 is still active using mamaInternal_findPayload from mamaMsg_destroy. Therefore the destroy calls can intercept bridge specific calls before they are made which would otherwise cause a crash.

The mama internal lock may also need to be opened up to avoid race conditions.

You can do something similar in Java finalizer methods via a JNI method.

Alternatively you could have a mama property which will simply prevent the payload bridges from being deallocated and will instead only destroy the middleware bridges. This way it will do what you want (i.e. always leak memory), but not be turned on for everyone unless they want it.

Cheers,

Frank

On Thu, Dec 28, 2017 at 4:25 PM, Bill Torpey <wallstprog@...> wrote:

See comments inline …

On Dec 28, 2017, at 11:07 AM, Frank Quinn <frank@...> wrote:

Hi Bill,

We close the bridges because we leak memory if we dont.

No, we shut down the bridges to prevent events from firing.  In some cases (and certainly *not* all) that also releases resources acquired when the bridges were opened, which is a Good Thing.

Imho leaking memory should be avoided where possible. In this case, its avoidable, so I don't think we should do it.

It’s not up to you.  As I said before, it’s my application, and I get to decide what leaks are worth worrying about.  One-time leaks that don’t grow over time don’t concern me, and the cost of eliminating them is not worth it in almost all cases.  (Again, that’s my choice — “Perfect is the enemy of good enough”).

If you really want control and are happy to leak memory, just don't call mama_close(),

As you are well aware, that is not a solution — it doesn’t stop events from firing, which is the real reason for shutting down the transport.

or we can look at that configuration option I suggested (though it will need to avoid freeing the bridges too - not just avoid dlclosing).

What configuration option is that?  How would it work?  

In terms of the actual fix, as I said, I haven't gotten around to it yet. I know how to fix it, I just haven't had the chance, nor have any volunteers come forward to pick it up.

I would be happy to submit a PR, but it doesn’t sound like it would be accepted.  If we can agree on an approach I’ll see what I can come up with.

In the meantime I gave up and forked the code.  As project maintainer I don’t imagine that’s what you want, but you’re not giving me a choice.

Cheers,

Frank

On 28 Dec 2017 15:49, "Bill Torpey" <wallstprog@...> wrote:

See comments inline …

On Dec 28, 2017, at 11:07 AM, Frank Quinn <frank@...> wrote:

Hi Bill,

We close the bridges because we leak memory if we dont.

No, we shut down the bridges to prevent events from firing.  In some cases (and certainly *not* all) that also releases resources acquired when the bridges were opened, which is a Good Thing.

Imho leaking memory should be avoided where possible. In this case, its avoidable, so I don't think we should do it.

It’s not up to you.  As I said before, it’s my application, and I get to decide what leaks are worth worrying about.  One-time leaks that don’t grow over time don’t concern me, and the cost of eliminating them is not worth it in almost all cases.  (Again, that’s my choice — “Perfect is the enemy of good enough”).

If you really want control and are happy to leak memory, just don't call mama_close(),

As you are well aware, that is not a solution — it doesn’t stop events from firing, which is the real reason for shutting down the transport.

or we can look at that configuration option I suggested (though it will need to avoid freeing the bridges too - not just avoid dlclosing).

What configuration option is that?  How would it work?  

In terms of the actual fix, as I said, I haven't gotten around to it yet. I know how to fix it, I just haven't had the chance, nor have any volunteers come forward to pick it up.

I would be happy to submit a PR, but it doesn’t sound like it would be accepted.  If we can agree on an approach I’ll see what I can come up with.

In the meantime I gave up and forked the code.  As project maintainer I don’t imagine that’s what you want, but you’re not giving me a choice.

Cheers,

Frank

On 28 Dec 2017 15:49, "Bill Torpey" <wallstprog@...> wrote:

HI Frank:

And Happy Holidays to you too!

A couple of points:

• I’m not aware of any changes to OpenMAMA that affect the issue, so if there are some please point me in the proper direction.  The last reply from you was back in May.
• As the application developer I get to decide which leaks are acceptable to me, and which are not.  Taking that choice away from me is not OK.
• You’re confusing and/or conflating the shutdown of the bridge libraries with the unloading of those libraries from memory.  
• It is not necessary to unload the libraries in order to shut them down:
• The transport library can be shut down without being unloaded from memory.
• The payload library doesn’t need to be shut down at all, since it never fires any events.
• The only reason I can think of for dynamically unloading the libraries is to support some kind of dynamic switching of transports and/or payloads.  I suspect that this feature is an example of YAGNI, but even there is a reasonable use for this feature, forcing the vast majority of applications that don’t need it to pay the price for it is a bad design decision.

Short version, it’s my application, and I get to decide how I want it to behave. 

Best Regards,

Bill 

On Dec 28, 2017, at 10:33 AM, Frank Quinn <frank@...> wrote:

Happy holidays folks!

First of all (with respect to the C++ concerns), that ticket is still open - I plan on actioning it I just haven't had time yet.

My suggestions were far from "don't solve it" and was instead was more like "let's not annoy every developer of OpenMAMA by leaking memory every single time they close their application" which is what was suggested. My opinion was that if there was an alternative, we should do that. If there was no alternative, we can reassess. Fortunately in this case there is an alternative since it's possible (thanks to last year's bridge changes) to programatically check if a specific bridge is still open in the finalizer / destructor and therefore not attempt to access the bridge if it has been unloaded. This is clean, unobtrusive and lightweight.

I also suggested a configuration option to optionally leave the payload bridge open (though as mentioned in the ticket if its memory is tied to the middleware bridge, it could crash anyway).

I would suggest a similar approach in Java - let the language specific layer deal with the language specific nuances. We can avoid crashes with code changes in OpenMAMA here fairly easily.

Cheers,

Frank

On Thu, Dec 28, 2017 at 3:17 PM, Sanjeev Wahi <sawahi@...> wrote:

I can suggest a possible fix (by adding extra weak_ptr check) while calling Mama.close()  that can avoid this problem in C11/C11++ when using shared_ptr.
*(assumption is Mama.close() is not called by many threads same time, in that case also use C11 atomic integer counter with this code)

*( I do not know much Java but something similar would work).



1st Approach:
Gat a new shared_ptr, but test for whether it is empty or pointing to something by testing
it for true/false, analogous to what we would do with a built-in pointer that might be zero:

void do_it(weak_ptr<Thing> wp){
        shared_ptr<Thing> sp = wp.lock(); // get shared_ptr from weak_ptr
        if(sp)
                sp->defrangulate(); // tell the Thing to do something
        else
                cout << "The Thing is gone!" << endl;
}



2nd Approach:
We can ask the weak_ptr if it has "expired":

bool is_it_there(weak_ptr<Thing> wp) {
        if(wp.expired()) {
                cout << "The Thing is gone!" << endl;
                return false;
        }
return true;
}





-Sanjeev Wahi

-----Original Message-----
From: openmama-dev-bounces@...nmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Bill Torpey
Sent: Thursday, December 28, 2017 9:39 AM
To: Yury Batrakov <yury.batrakov@...>
Cc: openmama-dev <openmama-dev@...rg>
Subject: Re: [Openmama-dev] Finalizers are dangerous

Unfortunately, that is not a bug, but a “feature”.

The problem is that mama_close unloads both the transport and payload libraries (via dlclose on Linux).  So, any access to any objects related to either library following mama_close is guaranteed to crash.

This makes OpenMAMA completely unusable for GC languages like Java, and presumably .Net, as well as for reference-counted implementations in other languages (e.g., shared_ptr in C++).

I’ve argued this point with Frank, but so far to no avail:  https://github.com/OpenMAMA/OpenMAMA/issues/264

Perhaps if enough people chime in, we can change Frank’s mind.  Until that time, the only solution I can think of is to fork OpenMAMA and remove or replace the offending code.  That is not a great solution, but as I mention in the bug report, this behavior is a total non-starter for me (and presumably for others as well).


> On Dec 28, 2017, at 6:08 AM, Yury Batrakov <yury.batrakov@...> wrote:
>
> Classification: Public
> Hi team,
>
> Sorry for telling bad news in holidays but I have found a major issue with Java API - JVM may crash if GC comes after Mama.close() method. Here's code sample to reproduce it:
>
> import com.wombat.mama.Mama;
> import com.wombat.mama.MamaMsg;
>
> public class Test {
>    private static MamaMsg getMessage() {
>        return new MamaMsg();
>    }
>
>    public static void main(String[] args) {
>        Mama.loadBridge("...");
>        Mama.open();
>
>        getMessage(); // Creating MamaMsg object without reference
>
>        Mama.close(); // Payload bridge is destroyed here
>        System.gc();
>        System.runFinalization(); // Calling MamaMsg.destroy() which delegates the destruction to deleted payload bridge
>    }
> }
>
> Stack trace:
> #12 0x00007fc494a095f0 in ?? ()
> #13 0x00007fc496ac1cf4 in mamaMsg_destroy (msg=0x7fc4900c90a0) at mama/c_cpp/src/c/msg.c:127
> #14 0x00007fc496d70b7f in Java_com_wombat_mama_MamaMsg__1destroy (env=0x7fc4b00039f8, this=0x7fc49779d710) at mama/jni/src/c/mamamsgjni.c:3882
> #15 0x00007fc4bae7e494 in ?? ()
>
> Problematic frame:
> #13 0x00007fc496ac1cf4 in mamaMsg_destroy (msg=0x7fc4900c90a0) at mama/c_cpp/src/c/msg.c:127
> 127             if (MAMA_STATUS_OK != impl->mPayloadBridge->msgPayloadDestroy (impl->mPayload))
>
> impl->mPayloadBridge is destroyed here.
>
> Similar crash occurs when finalizing subscriptions - they also need entitlements bridge to be available but Mama.close() deletes it too.
>
> The workaround is to call destroy() method for each message/subscription created but this actually nullifies the need for finalize() methods. I would delete all them from MAMA code.
>
>
> ---
> This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
>
> Please refer to https://www.db.com/disclosures for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.
> _______________________________________________
> Openmama-dev mailing list
> Openmama-dev@...g
> https://lists.openmama.org/mailman/listinfo/openmama-dev

_______________________________________________
Openmama-dev mailing list
Openmama-dev@...g
https://lists.openmama.org/mailman/listinfo/openmama-dev

_______________________________________________
Openmama-dev mailing list
Openmama-dev@...g
https://lists.openmama.org/mailman/listinfo/openmama-dev