Handling REFRESH messages in DQ publisher manager


Yury Batrakov
 

Classification: Public

Hi team,

I'm implementing new OpenMAMA transport bridge and have a question about Advanced publishing. In my middleware a publisher keeps track of all subscribers connected to it. A publisher and subscribers exchange with heartbeat messages posted to a single topic providing a string with session id in heartbeats' payload. When a publisher receives a heartbeat it sends a reply with the same session ID. Once heartbeat is not received by publisher it stops sending data and cleans up the resources allocated for a subscriber.
I tried implementing heartbeats with MAMA_SUBSC_REFRESH messages but the following problem occurs: in dqpublishermanager.c there is a handler for REFRESHES

switch (msgType)
{
case MAMA_SUBSC_REFRESH:
if (!impl->mRefreshResponseMsg)
{
*Here MAMA creates empty reply message*
mamaMsg_create(&impl->mRefreshResponseMsg);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_MISC);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgType.mFid, MAMA_MSG_TYPE_REFRESH);

}
*And sends it via the bridge*
mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
impl->mUserCallbacks.onRefresh((mamaDQPublisherManager)impl,
info, subType, msgType, msg );


The problem is that in the bridge we cannot restore the context of communication i.e. session id as mRefreshResponseMsg doesn't have any fields set. Thus we force users to implement a part of underlying protocol by handling refreshes themselves.

I'd propose the following change to fix that:

- mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
+ mamaDQPublisher_sendReply(info->pub, msg, impl->mRefreshResponseMsg);

Or could you advise how to workaround it?

Thanks,
Yury.


---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.


Ian Bell
 

Hi Yury

The MAMA level refreshes are generally separate from the middleware heartbeat functionality. Specially they operate at a higher level although they can be similar in nature.

Normal operation would be to have the transport clean up the client context information as the clients went away using its own heartbeat functionality and let MAMA inform the client application of subscribers using its own refresh messages. The refresh timeout is an hour so this is generally a lot higher than a transport heartbeat timeout.

The dqpublisher specifically does a send rather than a sendReply to reduce the heartbeat traffic as well as to allow for transports which don't maintain subscriber lists.

I.e. If you have 4 clients on 1 symbol each will send a refresh message once an hour less a random period of time, however the client refresh timeout is reset on receiving a refresh response from a publisher, so what actually happens is that the first client to timeout sends a refresh and then the rest of the clients reset their timeouts on the refresh response from the publisher. So the publisher only processes a single refresh message per topic as opposed to 1 for every client on every topic.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk

-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury Batrakov
Sent: 14 April 2015 13:43
To: openmama-dev@lists.openmama.org
Subject: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

Classification: Public

Hi team,

I'm implementing new OpenMAMA transport bridge and have a question about Advanced publishing. In my middleware a publisher keeps track of all subscribers connected to it. A publisher and subscribers exchange with heartbeat messages posted to a single topic providing a string with session id in heartbeats' payload. When a publisher receives a heartbeat it sends a reply with the same session ID. Once heartbeat is not received by publisher it stops sending data and cleans up the resources allocated for a subscriber.
I tried implementing heartbeats with MAMA_SUBSC_REFRESH messages but the following problem occurs: in dqpublishermanager.c there is a handler for REFRESHES

switch (msgType)
{
case MAMA_SUBSC_REFRESH:
if (!impl->mRefreshResponseMsg)
{
*Here MAMA creates empty reply message*
mamaMsg_create(&impl->mRefreshResponseMsg);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_MISC);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgType.mFid, MAMA_MSG_TYPE_REFRESH);

}
*And sends it via the bridge*
mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
impl->mUserCallbacks.onRefresh((mamaDQPublisherManager)impl,
info, subType, msgType, msg );


The problem is that in the bridge we cannot restore the context of communication i.e. session id as mRefreshResponseMsg doesn't have any fields set. Thus we force users to implement a part of underlying protocol by handling refreshes themselves.

I'd propose the following change to fix that:

- mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
+ mamaDQPublisher_sendReply(info->pub, msg, impl->mRefreshResponseMsg);

Or could you advise how to workaround it?

Thanks,
Yury.


---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev


Yury Batrakov
 

Classification: Public

Hi Ian,

Thanks for the explanation. Will do that on the application level.

-----Original Message-----
From: Ian Bell [mailto:ian.bell@hapsoftconsulting.co.uk]
Sent: Tuesday, April 14, 2015 4:34 PM
To: Yury Batrakov; openmama-dev@lists.openmama.org
Subject: RE: Handling REFRESH messages in DQ publisher manager

Hi Yury

The MAMA level refreshes are generally separate from the middleware heartbeat functionality. Specially they operate at a higher level although they can be similar in nature.

Normal operation would be to have the transport clean up the client context information as the clients went away using its own heartbeat functionality and let MAMA inform the client application of subscribers using its own refresh messages. The refresh timeout is an hour so this is generally a lot higher than a transport heartbeat timeout.

The dqpublisher specifically does a send rather than a sendReply to reduce the heartbeat traffic as well as to allow for transports which don't maintain subscriber lists.

I.e. If you have 4 clients on 1 symbol each will send a refresh message once an hour less a random period of time, however the client refresh timeout is reset on receiving a refresh response from a publisher, so what actually happens is that the first client to timeout sends a refresh and then the rest of the clients reset their timeouts on the refresh response from the publisher. So the publisher only processes a single refresh message per topic as opposed to 1 for every client on every topic.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk



-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury Batrakov
Sent: 14 April 2015 13:43
To: openmama-dev@lists.openmama.org
Subject: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

Classification: Public

Hi team,

I'm implementing new OpenMAMA transport bridge and have a question about Advanced publishing. In my middleware a publisher keeps track of all subscribers connected to it. A publisher and subscribers exchange with heartbeat messages posted to a single topic providing a string with session id in heartbeats' payload. When a publisher receives a heartbeat it sends a reply with the same session ID. Once heartbeat is not received by publisher it stops sending data and cleans up the resources allocated for a subscriber.
I tried implementing heartbeats with MAMA_SUBSC_REFRESH messages but the following problem occurs: in dqpublishermanager.c there is a handler for REFRESHES

switch (msgType)
{
case MAMA_SUBSC_REFRESH:
if (!impl->mRefreshResponseMsg)
{
*Here MAMA creates empty reply message*
mamaMsg_create(&impl->mRefreshResponseMsg);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_MISC);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgType.mFid, MAMA_MSG_TYPE_REFRESH);

}
*And sends it via the bridge*
mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
impl->mUserCallbacks.onRefresh((mamaDQPublisherManager)impl,
info, subType, msgType, msg );


The problem is that in the bridge we cannot restore the context of communication i.e. session id as mRefreshResponseMsg doesn't have any fields set. Thus we force users to implement a part of underlying protocol by handling refreshes themselves.

I'd propose the following change to fix that:

- mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
+ mamaDQPublisher_sendReply(info->pub, msg, impl->mRefreshResponseMsg);

Or could you advise how to workaround it?

Thanks,
Yury.


---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev


---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.


Glenn McClements <g.mcclements@...>
 

Thanks Ian, you beat me to it!

First of all Yury, welcome to the list and thanks for your question.

As Ian mentions one of the issues here is that you are implementing a middleware level mechanism using OpenMAMA messages, so the solution is not to modify dqpublishermanager to handle the new use of REFRESHs, but to make sure that whatever messages are used for heartbeats are never passed up to the core OpenMAMA level.


I’d also be against adding a new HEARTBEAT OpenMAMA message type for this use case as would imply that heartbeats would be sent/received/processed the OpenMAMA level (which is something we may add at some point) and *not* at the middleware level. That’s not to say you can’t add a new specific message type for your own heartbeats, but it should be middleware/bridge specific.

Also a minor point is that mamaDQPublisher_sendReply() to me implies replying to an mamaInbox, and for refreshes there is (currently) no inbox to reply to.

Thanks,
Glenn

On 14/04/2015 14:34, "Ian Bell" <ian.bell@hapsoftconsulting.co.uk> wrote:

Hi Yury

The MAMA level refreshes are generally separate from the middleware heartbeat functionality. Specially they operate at a higher level although they can be similar in nature.

Normal operation would be to have the transport clean up the client context information as the clients went away using its own heartbeat functionality and let MAMA inform the client application of subscribers using its own refresh messages. The refresh timeout is an hour so this is generally a lot higher than a transport heartbeat timeout.

The dqpublisher specifically does a send rather than a sendReply to reduce the heartbeat traffic as well as to allow for transports which don't maintain subscriber lists.

I.e. If you have 4 clients on 1 symbol each will send a refresh message once an hour less a random period of time, however the client refresh timeout is reset on receiving a refresh response from a publisher, so what actually happens is that the first client to timeout sends a refresh and then the rest of the clients reset their timeouts on the refresh response from the publisher. So the publisher only processes a single refresh message per topic as opposed to 1 for every client on every topic.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk



-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury Batrakov
Sent: 14 April 2015 13:43
To: openmama-dev@lists.openmama.org
Subject: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

Classification: Public

Hi team,

I'm implementing new OpenMAMA transport bridge and have a question about Advanced publishing. In my middleware a publisher keeps track of all subscribers connected to it. A publisher and subscribers exchange with heartbeat messages posted to a single topic providing a string with session id in heartbeats' payload. When a publisher receives a heartbeat it sends a reply with the same session ID. Once heartbeat is not received by publisher it stops sending data and cleans up the resources allocated for a subscriber.
I tried implementing heartbeats with MAMA_SUBSC_REFRESH messages but the following problem occurs: in dqpublishermanager.c there is a handler for REFRESHES

switch (msgType)
{
case MAMA_SUBSC_REFRESH:
if (!impl->mRefreshResponseMsg)
{
*Here MAMA creates empty reply message*
mamaMsg_create(&impl->mRefreshResponseMsg);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_MISC);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgType.mFid, MAMA_MSG_TYPE_REFRESH);

}
*And sends it via the bridge*
mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
impl->mUserCallbacks.onRefresh((mamaDQPublisherManager)impl,
info, subType, msgType, msg );


The problem is that in the bridge we cannot restore the context of communication i.e. session id as mRefreshResponseMsg doesn't have any fields set. Thus we force users to implement a part of underlying protocol by handling refreshes themselves.

I'd propose the following change to fix that:

- mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
+ mamaDQPublisher_sendReply(info->pub, msg, impl->mRefreshResponseMsg);

Or could you advise how to workaround it?

Thanks,
Yury.


---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev


Yury Batrakov
 

Classification: Public

Thanks Glenn,

BTW what is best practice of implementing custom messages? I mean what is to return for mamaMsg_get(MdMsgType, 1), any integer which is not in standard list of update types?

-----Original Message-----
From: Glenn McClements [mailto:g.mcclements@srtechlabs.com]
Sent: Tuesday, April 14, 2015 4:53 PM
To: Ian Bell; Yury Batrakov; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

Thanks Ian, you beat me to it!

First of all Yury, welcome to the list and thanks for your question.

As Ian mentions one of the issues here is that you are implementing a middleware level mechanism using OpenMAMA messages, so the solution is not to modify dqpublishermanager to handle the new use of REFRESHs, but to make sure that whatever messages are used for heartbeats are never passed up to the core OpenMAMA level.


I’d also be against adding a new HEARTBEAT OpenMAMA message type for this use case as would imply that heartbeats would be sent/received/processed the OpenMAMA level (which is something we may add at some point) and *not* at the middleware level. That’s not to say you can’t add a new specific message type for your own heartbeats, but it should be middleware/bridge specific.

Also a minor point is that mamaDQPublisher_sendReply() to me implies replying to an mamaInbox, and for refreshes there is (currently) no inbox to reply to.

Thanks,
Glenn

On 14/04/2015 14:34, "Ian Bell" <ian.bell@hapsoftconsulting.co.uk> wrote:

Hi Yury

The MAMA level refreshes are generally separate from the middleware heartbeat functionality. Specially they operate at a higher level although they can be similar in nature.

Normal operation would be to have the transport clean up the client context information as the clients went away using its own heartbeat functionality and let MAMA inform the client application of subscribers using its own refresh messages. The refresh timeout is an hour so this is generally a lot higher than a transport heartbeat timeout.

The dqpublisher specifically does a send rather than a sendReply to reduce the heartbeat traffic as well as to allow for transports which don't maintain subscriber lists.

I.e. If you have 4 clients on 1 symbol each will send a refresh message once an hour less a random period of time, however the client refresh timeout is reset on receiving a refresh response from a publisher, so what actually happens is that the first client to timeout sends a refresh and then the rest of the clients reset their timeouts on the refresh response from the publisher. So the publisher only processes a single refresh message per topic as opposed to 1 for every client on every topic.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk



-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org
[mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury
Batrakov
Sent: 14 April 2015 13:43
To: openmama-dev@lists.openmama.org
Subject: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Hi team,

I'm implementing new OpenMAMA transport bridge and have a question about Advanced publishing. In my middleware a publisher keeps track of all subscribers connected to it. A publisher and subscribers exchange with heartbeat messages posted to a single topic providing a string with session id in heartbeats' payload. When a publisher receives a heartbeat it sends a reply with the same session ID. Once heartbeat is not received by publisher it stops sending data and cleans up the resources allocated for a subscriber.
I tried implementing heartbeats with MAMA_SUBSC_REFRESH messages but
the following problem occurs: in dqpublishermanager.c there is a
handler for REFRESHES

switch (msgType)
{
case MAMA_SUBSC_REFRESH:
if (!impl->mRefreshResponseMsg)
{
*Here MAMA creates empty reply message*
mamaMsg_create(&impl->mRefreshResponseMsg);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_MISC);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgType.mFid,
MAMA_MSG_TYPE_REFRESH);

}
*And sends it via the bridge*
mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
impl->mUserCallbacks.onRefresh((mamaDQPublisherManager)impl,
info, subType,
msgType, msg );


The problem is that in the bridge we cannot restore the context of communication i.e. session id as mRefreshResponseMsg doesn't have any fields set. Thus we force users to implement a part of underlying protocol by handling refreshes themselves.

I'd propose the following change to fix that:

- mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
+ mamaDQPublisher_sendReply(info->pub, msg, impl->mRefreshResponseMsg);

Or could you advise how to workaround it?

Thanks,
Yury.


---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev

---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.


Alireza Assadzadeh <Alireza.Assadzadeh@...>
 

BTW, I noticed the following description for MAMA_MSG_TYPE_NULL in mamaMsgType. I did not see this message type having special logic associated with it in OpenMAMA MAMC/MAMACPP.

From msgtype.h:

/** Keep alive message. */
MAMA_MSG_TYPE_NULL = 175,

Is this message type suitable for this case?

--Alireza

-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-
dev-bounces@lists.openmama.org] On Behalf Of Yury Batrakov
Sent: Tuesday, April 14, 2015 11:03 AM
To: Glenn McClements; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Thanks Glenn,

BTW what is best practice of implementing custom messages? I mean what is
to return for mamaMsg_get(MdMsgType, 1), any integer which is not in
standard list of update types?


-----Original Message-----
From: Glenn McClements [mailto:g.mcclements@srtechlabs.com]
Sent: Tuesday, April 14, 2015 4:53 PM
To: Ian Bell; Yury Batrakov; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Thanks Ian, you beat me to it!

First of all Yury, welcome to the list and thanks for your question.

As Ian mentions one of the issues here is that you are implementing a
middleware level mechanism using OpenMAMA messages, so the solution is
not to modify dqpublishermanager to handle the new use of REFRESHs, but
to make sure that whatever messages are used for heartbeats are never
passed up to the core OpenMAMA level.


I’d also be against adding a new HEARTBEAT OpenMAMA message type for
this use case as would imply that heartbeats would be
sent/received/processed the OpenMAMA level (which is something we may
add at some point) and *not* at the middleware level. That’s not to say you
can’t add a new specific message type for your own heartbeats, but it should
be middleware/bridge specific.

Also a minor point is that mamaDQPublisher_sendReply() to me implies
replying to an mamaInbox, and for refreshes there is (currently) no inbox to
reply to.

Thanks,
Glenn

On 14/04/2015 14:34, "Ian Bell" <ian.bell@hapsoftconsulting.co.uk> wrote:

Hi Yury

The MAMA level refreshes are generally separate from the middleware
heartbeat functionality. Specially they operate at a higher level although
they can be similar in nature.

Normal operation would be to have the transport clean up the client
context information as the clients went away using its own heartbeat
functionality and let MAMA inform the client application of subscribers using
its own refresh messages. The refresh timeout is an hour so this is generally
a lot higher than a transport heartbeat timeout.

The dqpublisher specifically does a send rather than a sendReply to reduce
the heartbeat traffic as well as to allow for transports which don't maintain
subscriber lists.

I.e. If you have 4 clients on 1 symbol each will send a refresh message once
an hour less a random period of time, however the client refresh timeout is
reset on receiving a refresh response from a publisher, so what actually
happens is that the first client to timeout sends a refresh and then the rest of
the clients reset their timeouts on the refresh response from the publisher.
So the publisher only processes a single refresh message per topic as
opposed to 1 for every client on every topic.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk



-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org
[mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury
Batrakov
Sent: 14 April 2015 13:43
To: openmama-dev@lists.openmama.org
Subject: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Hi team,

I'm implementing new OpenMAMA transport bridge and have a question
about Advanced publishing. In my middleware a publisher keeps track of all
subscribers connected to it. A publisher and subscribers exchange with
heartbeat messages posted to a single topic providing a string with session id
in heartbeats' payload. When a publisher receives a heartbeat it sends a
reply with the same session ID. Once heartbeat is not received by publisher it
stops sending data and cleans up the resources allocated for a subscriber.
I tried implementing heartbeats with MAMA_SUBSC_REFRESH messages
but
the following problem occurs: in dqpublishermanager.c there is a
handler for REFRESHES

switch (msgType)
{
case MAMA_SUBSC_REFRESH:
if (!impl->mRefreshResponseMsg)
{
*Here MAMA creates empty reply message*
mamaMsg_create(&impl->mRefreshResponseMsg);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_MISC);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgType.mFid,
MAMA_MSG_TYPE_REFRESH);

}
*And sends it via the bridge*
mamaDQPublisher_send(info->pub, impl-
mRefreshResponseMsg);
impl-
mUserCallbacks.onRefresh((mamaDQPublisherManager)impl,
info, subType,
msgType, msg );


The problem is that in the bridge we cannot restore the context of
communication i.e. session id as mRefreshResponseMsg doesn't have any
fields set. Thus we force users to implement a part of underlying protocol by
handling refreshes themselves.

I'd propose the following change to fix that:

- mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
+ mamaDQPublisher_sendReply(info->pub, msg, impl-
mRefreshResponseMsg);

Or could you advise how to workaround it?

Thanks,
Yury.


---
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error) please
notify the sender immediately and delete this e-mail. Any unauthorized
copying, disclosure or distribution of the material in this e-mail is strictly
forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures and to
http://www.db.com/unitedkingdom/content/privacy.htm for information
about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev

---
This e-mail may contain confidential and/or privileged information. If you are
not the intended recipient (or have received this e-mail in error) please
notify the sender immediately and delete this e-mail. Any unauthorized
copying, disclosure or distribution of the material in this e-mail is strictly
forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures and to
http://www.db.com/unitedkingdom/content/privacy.htm for information
about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev


Glenn McClements <g.mcclements@...>
 

Actually typically OpenMAMA messages are either:
- always passed back back the application
Or
- for a small amount of types such as REFRESH, handled in the core OpenMAMA code.

Other message types are dealt with at the application level, as OpenMAMA is dumb with them and simple passes them on up, however you are asking for something different in that a message that shouldn’t really progress past the bridge/middleware level.

I can’t think of any implementation I’m familiar with that uses an OpenMAMA message type at the bridge/middleware level only but it’s something to think about a bit more. Possibly a new MAMA_BRIDGE_INTERNAL type would be required.There would also be a question to whether the messages terminate in the bridge/middleware level or the OpenMAMA level - the user application should never see these.

Other middlewares typically have their own internal messages types for the kind of use case you mention below so OpenMAMA messages never get to see them.

Regards,
Glenn

On 14/04/2015 16:02, "Yury Batrakov" <yury.batrakov@db.com> wrote:

Classification: Public

Thanks Glenn,

BTW what is best practice of implementing custom messages? I mean what is to return for mamaMsg_get(MdMsgType, 1), any integer which is not in standard list of update types?


-----Original Message-----
From: Glenn McClements [mailto:g.mcclements@srtechlabs.com]
Sent: Tuesday, April 14, 2015 4:53 PM
To: Ian Bell; Yury Batrakov; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

Thanks Ian, you beat me to it!

First of all Yury, welcome to the list and thanks for your question.

As Ian mentions one of the issues here is that you are implementing a middleware level mechanism using OpenMAMA messages, so the solution is not to modify dqpublishermanager to handle the new use of REFRESHs, but to make sure that whatever messages are used for heartbeats are never passed up to the core OpenMAMA level.


I’d also be against adding a new HEARTBEAT OpenMAMA message type for this use case as would imply that heartbeats would be sent/received/processed the OpenMAMA level (which is something we may add at some point) and *not* at the middleware level. That’s not to say you can’t add a new specific message type for your own heartbeats, but it should be middleware/bridge specific.

Also a minor point is that mamaDQPublisher_sendReply() to me implies replying to an mamaInbox, and for refreshes there is (currently) no inbox to reply to.

Thanks,
Glenn

On 14/04/2015 14:34, "Ian Bell" <ian.bell@hapsoftconsulting.co.uk> wrote:

Hi Yury

The MAMA level refreshes are generally separate from the middleware heartbeat functionality. Specially they operate at a higher level although they can be similar in nature.

Normal operation would be to have the transport clean up the client context information as the clients went away using its own heartbeat functionality and let MAMA inform the client application of subscribers using its own refresh messages. The refresh timeout is an hour so this is generally a lot higher than a transport heartbeat timeout.

The dqpublisher specifically does a send rather than a sendReply to reduce the heartbeat traffic as well as to allow for transports which don't maintain subscriber lists.

I.e. If you have 4 clients on 1 symbol each will send a refresh message once an hour less a random period of time, however the client refresh timeout is reset on receiving a refresh response from a publisher, so what actually happens is that the first client to timeout sends a refresh and then the rest of the clients reset their timeouts on the refresh response from the publisher. So the publisher only processes a single refresh message per topic as opposed to 1 for every client on every topic.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk



-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org
[mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury
Batrakov
Sent: 14 April 2015 13:43
To: openmama-dev@lists.openmama.org
Subject: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Hi team,

I'm implementing new OpenMAMA transport bridge and have a question about Advanced publishing. In my middleware a publisher keeps track of all subscribers connected to it. A publisher and subscribers exchange with heartbeat messages posted to a single topic providing a string with session id in heartbeats' payload. When a publisher receives a heartbeat it sends a reply with the same session ID. Once heartbeat is not received by publisher it stops sending data and cleans up the resources allocated for a subscriber.
I tried implementing heartbeats with MAMA_SUBSC_REFRESH messages but
the following problem occurs: in dqpublishermanager.c there is a
handler for REFRESHES

switch (msgType)
{
case MAMA_SUBSC_REFRESH:
if (!impl->mRefreshResponseMsg)
{
*Here MAMA creates empty reply message*
mamaMsg_create(&impl->mRefreshResponseMsg);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_MISC);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgType.mFid,
MAMA_MSG_TYPE_REFRESH);

}
*And sends it via the bridge*
mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
impl->mUserCallbacks.onRefresh((mamaDQPublisherManager)impl,
info, subType,
msgType, msg );


The problem is that in the bridge we cannot restore the context of communication i.e. session id as mRefreshResponseMsg doesn't have any fields set. Thus we force users to implement a part of underlying protocol by handling refreshes themselves.

I'd propose the following change to fix that:

- mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
+ mamaDQPublisher_sendReply(info->pub, msg, impl->mRefreshResponseMsg);

Or could you advise how to workaround it?

Thanks,
Yury.


---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev

---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.


Ian Bell
 

Hi

I would have to agree with Glenn here, it seems that these kind of heartbeat messages should be internal to the bridge and not passed up to MAMA and as such probably should not be represented in the MsgType enum.

In the past I've seen heartbeat messages and this type of event handled within the bridge and ending up in transport level callbacks.
Eg heartbeats from a publisher to the subscriber can cause a disconnect callback to be invoked when the heartbeats cease.
Another example is the Topic callbacks which exist on the transport and can be used to show when a subscriber for a symbol appears and disappears.
Both of these cases are triggered by transport messages but the msg is not visible or passed up through the MAMA layer.

Thanks,
Ian

-----Original Message-----
From: Glenn McClements [mailto:g.mcclements@srtechlabs.com]
Sent: 14 April 2015 18:28
To: Yury Batrakov; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

Actually typically OpenMAMA messages are either:
- always passed back back the application Or
- for a small amount of types such as REFRESH, handled in the core OpenMAMA code.

Other message types are dealt with at the application level, as OpenMAMA is dumb with them and simple passes them on up, however you are asking for something different in that a message that shouldn’t really progress past the bridge/middleware level.

I can’t think of any implementation I’m familiar with that uses an OpenMAMA message type at the bridge/middleware level only but it’s something to think about a bit more. Possibly a new MAMA_BRIDGE_INTERNAL type would be required.There would also be a question to whether the messages terminate in the bridge/middleware level or the OpenMAMA level - the user application should never see these.

Other middlewares typically have their own internal messages types for the kind of use case you mention below so OpenMAMA messages never get to see them.

Regards,
Glenn



On 14/04/2015 16:02, "Yury Batrakov" <yury.batrakov@db.com> wrote:

Classification: Public

Thanks Glenn,

BTW what is best practice of implementing custom messages? I mean what is to return for mamaMsg_get(MdMsgType, 1), any integer which is not in standard list of update types?


-----Original Message-----
From: Glenn McClements [mailto:g.mcclements@srtechlabs.com]
Sent: Tuesday, April 14, 2015 4:53 PM
To: Ian Bell; Yury Batrakov; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Thanks Ian, you beat me to it!

First of all Yury, welcome to the list and thanks for your question.

As Ian mentions one of the issues here is that you are implementing a middleware level mechanism using OpenMAMA messages, so the solution is not to modify dqpublishermanager to handle the new use of REFRESHs, but to make sure that whatever messages are used for heartbeats are never passed up to the core OpenMAMA level.


I’d also be against adding a new HEARTBEAT OpenMAMA message type for this use case as would imply that heartbeats would be sent/received/processed the OpenMAMA level (which is something we may add at some point) and *not* at the middleware level. That’s not to say you can’t add a new specific message type for your own heartbeats, but it should be middleware/bridge specific.

Also a minor point is that mamaDQPublisher_sendReply() to me implies replying to an mamaInbox, and for refreshes there is (currently) no inbox to reply to.

Thanks,
Glenn

On 14/04/2015 14:34, "Ian Bell" <ian.bell@hapsoftconsulting.co.uk> wrote:

Hi Yury

The MAMA level refreshes are generally separate from the middleware heartbeat functionality. Specially they operate at a higher level although they can be similar in nature.

Normal operation would be to have the transport clean up the client context information as the clients went away using its own heartbeat functionality and let MAMA inform the client application of subscribers using its own refresh messages. The refresh timeout is an hour so this is generally a lot higher than a transport heartbeat timeout.

The dqpublisher specifically does a send rather than a sendReply to reduce the heartbeat traffic as well as to allow for transports which don't maintain subscriber lists.

I.e. If you have 4 clients on 1 symbol each will send a refresh message once an hour less a random period of time, however the client refresh timeout is reset on receiving a refresh response from a publisher, so what actually happens is that the first client to timeout sends a refresh and then the rest of the clients reset their timeouts on the refresh response from the publisher. So the publisher only processes a single refresh message per topic as opposed to 1 for every client on every topic.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk



-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org
[mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury
Batrakov
Sent: 14 April 2015 13:43
To: openmama-dev@lists.openmama.org
Subject: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Hi team,

I'm implementing new OpenMAMA transport bridge and have a question about Advanced publishing. In my middleware a publisher keeps track of all subscribers connected to it. A publisher and subscribers exchange with heartbeat messages posted to a single topic providing a string with session id in heartbeats' payload. When a publisher receives a heartbeat it sends a reply with the same session ID. Once heartbeat is not received by publisher it stops sending data and cleans up the resources allocated for a subscriber.
I tried implementing heartbeats with MAMA_SUBSC_REFRESH messages but
the following problem occurs: in dqpublishermanager.c there is a
handler for REFRESHES

switch (msgType)
{
case MAMA_SUBSC_REFRESH:
if (!impl->mRefreshResponseMsg)
{
*Here MAMA creates empty reply message*
mamaMsg_create(&impl->mRefreshResponseMsg);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_MISC);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgType.mFid,
MAMA_MSG_TYPE_REFRESH);

}
*And sends it via the bridge*
mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
impl->mUserCallbacks.onRefresh((mamaDQPublisherManager)impl,
info, subType,
msgType, msg );


The problem is that in the bridge we cannot restore the context of communication i.e. session id as mRefreshResponseMsg doesn't have any fields set. Thus we force users to implement a part of underlying protocol by handling refreshes themselves.

I'd propose the following change to fix that:

- mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
+ mamaDQPublisher_sendReply(info->pub, msg,
+ impl->mRefreshResponseMsg);

Or could you advise how to workaround it?

Thanks,
Yury.


---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev

---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.


Tom Doust
 

Although I can think of many reasons why this would be a bad idea, it might be worth considering defining some MAMA_MSG_TYPE_PLATFORM

This would allow bridge developers to deliver custom stuff to client code with the intention of formalising into the api when it turns out to become useful.

An example of this would be the ACK/NAK messages our RMDS bridge delivers in response to message posts on TREP. We implemented these in the absence of any publisher response mechanism in OpenMAMA and when the new publisher callbacks that are currently under development appear in a version of OpenMAMA we will update the bridge to use those instead


Tom

-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Alireza Assadzadeh
Sent: 14 April 2015 17:06
To: Yury Batrakov; Glenn McClements; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

BTW, I noticed the following description for MAMA_MSG_TYPE_NULL in mamaMsgType. I did not see this message type having special logic associated with it in OpenMAMA MAMC/MAMACPP.

From msgtype.h:

/** Keep alive message. */
MAMA_MSG_TYPE_NULL = 175,

Is this message type suitable for this case?

--Alireza

-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-
dev-bounces@lists.openmama.org] On Behalf Of Yury Batrakov
Sent: Tuesday, April 14, 2015 11:03 AM
To: Glenn McClements; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Thanks Glenn,

BTW what is best practice of implementing custom messages? I mean what
is to return for mamaMsg_get(MdMsgType, 1), any integer which is not
in standard list of update types?


-----Original Message-----
From: Glenn McClements [mailto:g.mcclements@srtechlabs.com]
Sent: Tuesday, April 14, 2015 4:53 PM
To: Ian Bell; Yury Batrakov; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Thanks Ian, you beat me to it!

First of all Yury, welcome to the list and thanks for your question.

As Ian mentions one of the issues here is that you are implementing a
middleware level mechanism using OpenMAMA messages, so the solution is
not to modify dqpublishermanager to handle the new use of REFRESHs,
but to make sure that whatever messages are used for heartbeats are
never passed up to the core OpenMAMA level.


I’d also be against adding a new HEARTBEAT OpenMAMA message type for
this use case as would imply that heartbeats would be
sent/received/processed the OpenMAMA level (which is something we may
add at some point) and *not* at the middleware level. That’s not to
say you can’t add a new specific message type for your own heartbeats,
but it should be middleware/bridge specific.

Also a minor point is that mamaDQPublisher_sendReply() to me implies
replying to an mamaInbox, and for refreshes there is (currently) no
inbox to reply to.

Thanks,
Glenn

On 14/04/2015 14:34, "Ian Bell" <ian.bell@hapsoftconsulting.co.uk> wrote:

Hi Yury

The MAMA level refreshes are generally separate from the middleware
heartbeat functionality. Specially they operate at a higher level
although they can be similar in nature.

Normal operation would be to have the transport clean up the client
context information as the clients went away using its own heartbeat
functionality and let MAMA inform the client application of
subscribers using its own refresh messages. The refresh timeout is an
hour so this is generally a lot higher than a transport heartbeat timeout.

The dqpublisher specifically does a send rather than a sendReply to
reduce
the heartbeat traffic as well as to allow for transports which don't
maintain subscriber lists.

I.e. If you have 4 clients on 1 symbol each will send a refresh
message once
an hour less a random period of time, however the client refresh
timeout is reset on receiving a refresh response from a publisher, so
what actually happens is that the first client to timeout sends a
refresh and then the rest of the clients reset their timeouts on the refresh response from the publisher.
So the publisher only processes a single refresh message per topic as
opposed to 1 for every client on every topic.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk



-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org
[mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury
Batrakov
Sent: 14 April 2015 13:43
To: openmama-dev@lists.openmama.org
Subject: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Hi team,

I'm implementing new OpenMAMA transport bridge and have a question
about Advanced publishing. In my middleware a publisher keeps track of
all subscribers connected to it. A publisher and subscribers exchange
with heartbeat messages posted to a single topic providing a string
with session id in heartbeats' payload. When a publisher receives a
heartbeat it sends a reply with the same session ID. Once heartbeat is
not received by publisher it stops sending data and cleans up the resources allocated for a subscriber.
I tried implementing heartbeats with MAMA_SUBSC_REFRESH messages
but
the following problem occurs: in dqpublishermanager.c there is a
handler for REFRESHES

switch (msgType)
{
case MAMA_SUBSC_REFRESH:
if (!impl->mRefreshResponseMsg)
{
*Here MAMA creates empty reply message*
mamaMsg_create(&impl->mRefreshResponseMsg);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_MISC);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgType.mFid,
MAMA_MSG_TYPE_REFRESH);

}
*And sends it via the bridge*
mamaDQPublisher_send(info->pub, impl-
mRefreshResponseMsg);
impl-
mUserCallbacks.onRefresh((mamaDQPublisherManager)impl,
info, subType,
msgType, msg );


The problem is that in the bridge we cannot restore the context of
communication i.e. session id as mRefreshResponseMsg doesn't have any
fields set. Thus we force users to implement a part of underlying
protocol by handling refreshes themselves.

I'd propose the following change to fix that:

- mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
+ mamaDQPublisher_sendReply(info->pub, msg, impl-
mRefreshResponseMsg);

Or could you advise how to workaround it?

Thanks,
Yury.


---
This e-mail may contain confidential and/or privileged information.
If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and delete this e-mail. Any
unauthorized copying, disclosure or distribution of the material in
this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures and to
http://www.db.com/unitedkingdom/content/privacy.htm for information
about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev

---
This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient (or have received this e-mail in
error) please notify the sender immediately and delete this e-mail.
Any unauthorized copying, disclosure or distribution of the material
in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures and to
http://www.db.com/unitedkingdom/content/privacy.htm for information
about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev


Glenn McClements <g.mcclements@...>
 

Hi Tom,
I’m not sure what distinguishes the messages you suggest from any other kind of application messages, which don’t even need to have a message type. The ANK/NAK messages you suggest look like they’re dealt with at a custom block at the application level, so OpenMAMA just needs to be dumb in this respect and pass those messages though.

In summary the stack is the below:

1) Middleware
------------
2) Bridge

------------
3) OpenMAMA
-------------
4) Application



(Hopefully the above comes out the mail server formatted OK)

And the messages we’ve been talking about terminate at the levels:

1) Middleware heartbeats (generally) other internal middleware messages

2) Not too much currently terminates at this level, some middlewares pass back info messages or callbacks which may or may not be propagated up to OpenMAMA. Yesterday when I mentioned the possibility of adding a new INTERNAL message type for the use case mentioned, I was considering it terminating here, or possibly up to OpenMAMA which would ignore it.

3) Some OpenMAMA internal messages, such as REFRESHs.

4) Everything else, including NULL messages. Sorry Alireza, I missed your mail first time round

Heartbeats could theoretically live in any level, but For Yury’s use case it’s specifically 1 & 2, and therefore I don’t think these messages should be propagated up. NULL messages could be used, but then you would be adding expected behaviour onto the application level e.g. they should be ignored, and this is something I try to avoid.

I think there is room here to make this more explicit, possibly though a couple of new message types or namespaces but this could be complicating things.

Yury - it actually just occurred to me, are these heartbeats on a per transport, publisher or topic level? If they’re of the transport or publisher, then another possibility would be to have a separate middleware level subscription which you could send whatever you wanted over, and OpenMAMA wouldn’t be affected at all.

Regards,
Glenn

On 16/04/2015 10:21, "Tom Doust" <tom.doust@tick42.com> wrote:

Although I can think of many reasons why this would be a bad idea, it might be worth considering defining some MAMA_MSG_TYPE_PLATFORM

This would allow bridge developers to deliver custom stuff to client code with the intention of formalising into the api when it turns out to become useful.

An example of this would be the ACK/NAK messages our RMDS bridge delivers in response to message posts on TREP. We implemented these in the absence of any publisher response mechanism in OpenMAMA and when the new publisher callbacks that are currently under development appear in a version of OpenMAMA we will update the bridge to use those instead


Tom

-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Alireza Assadzadeh
Sent: 14 April 2015 17:06
To: Yury Batrakov; Glenn McClements; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

BTW, I noticed the following description for MAMA_MSG_TYPE_NULL in mamaMsgType. I did not see this message type having special logic associated with it in OpenMAMA MAMC/MAMACPP.

From msgtype.h:

/** Keep alive message. */
MAMA_MSG_TYPE_NULL = 175,

Is this message type suitable for this case?

--Alireza

-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-
dev-bounces@lists.openmama.org] On Behalf Of Yury Batrakov
Sent: Tuesday, April 14, 2015 11:03 AM
To: Glenn McClements; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Thanks Glenn,

BTW what is best practice of implementing custom messages? I mean what
is to return for mamaMsg_get(MdMsgType, 1), any integer which is not
in standard list of update types?


-----Original Message-----
From: Glenn McClements [mailto:g.mcclements@srtechlabs.com]
Sent: Tuesday, April 14, 2015 4:53 PM
To: Ian Bell; Yury Batrakov; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Thanks Ian, you beat me to it!

First of all Yury, welcome to the list and thanks for your question.

As Ian mentions one of the issues here is that you are implementing a
middleware level mechanism using OpenMAMA messages, so the solution is
not to modify dqpublishermanager to handle the new use of REFRESHs,
but to make sure that whatever messages are used for heartbeats are
never passed up to the core OpenMAMA level.


I’d also be against adding a new HEARTBEAT OpenMAMA message type for
this use case as would imply that heartbeats would be
sent/received/processed the OpenMAMA level (which is something we may
add at some point) and *not* at the middleware level. That’s not to
say you can’t add a new specific message type for your own heartbeats,
but it should be middleware/bridge specific.

Also a minor point is that mamaDQPublisher_sendReply() to me implies
replying to an mamaInbox, and for refreshes there is (currently) no
inbox to reply to.

Thanks,
Glenn

On 14/04/2015 14:34, "Ian Bell" <ian.bell@hapsoftconsulting.co.uk> wrote:

Hi Yury

The MAMA level refreshes are generally separate from the middleware
heartbeat functionality. Specially they operate at a higher level
although they can be similar in nature.

Normal operation would be to have the transport clean up the client
context information as the clients went away using its own heartbeat
functionality and let MAMA inform the client application of
subscribers using its own refresh messages. The refresh timeout is an
hour so this is generally a lot higher than a transport heartbeat timeout.

The dqpublisher specifically does a send rather than a sendReply to
reduce
the heartbeat traffic as well as to allow for transports which don't
maintain subscriber lists.

I.e. If you have 4 clients on 1 symbol each will send a refresh
message once
an hour less a random period of time, however the client refresh
timeout is reset on receiving a refresh response from a publisher, so
what actually happens is that the first client to timeout sends a
refresh and then the rest of the clients reset their timeouts on the refresh response from the publisher.
So the publisher only processes a single refresh message per topic as
opposed to 1 for every client on every topic.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk



-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org
[mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury
Batrakov
Sent: 14 April 2015 13:43
To: openmama-dev@lists.openmama.org
Subject: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Hi team,

I'm implementing new OpenMAMA transport bridge and have a question
about Advanced publishing. In my middleware a publisher keeps track of
all subscribers connected to it. A publisher and subscribers exchange
with heartbeat messages posted to a single topic providing a string
with session id in heartbeats' payload. When a publisher receives a
heartbeat it sends a reply with the same session ID. Once heartbeat is
not received by publisher it stops sending data and cleans up the resources allocated for a subscriber.
I tried implementing heartbeats with MAMA_SUBSC_REFRESH messages
but
the following problem occurs: in dqpublishermanager.c there is a
handler for REFRESHES

switch (msgType)
{
case MAMA_SUBSC_REFRESH:
if (!impl->mRefreshResponseMsg)
{
*Here MAMA creates empty reply message*
mamaMsg_create(&impl->mRefreshResponseMsg);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_MISC);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgType.mFid,
MAMA_MSG_TYPE_REFRESH);

}
*And sends it via the bridge*
mamaDQPublisher_send(info->pub, impl-
mRefreshResponseMsg);
impl-
mUserCallbacks.onRefresh((mamaDQPublisherManager)impl,
info, subType,
msgType, msg );


The problem is that in the bridge we cannot restore the context of
communication i.e. session id as mRefreshResponseMsg doesn't have any
fields set. Thus we force users to implement a part of underlying
protocol by handling refreshes themselves.

I'd propose the following change to fix that:

- mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
+ mamaDQPublisher_sendReply(info->pub, msg, impl-
mRefreshResponseMsg);

Or could you advise how to workaround it?

Thanks,
Yury.


---
This e-mail may contain confidential and/or privileged information.
If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and delete this e-mail. Any
unauthorized copying, disclosure or distribution of the material in
this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures and to
http://www.db.com/unitedkingdom/content/privacy.htm for information
about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev

---
This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient (or have received this e-mail in
error) please notify the sender immediately and delete this e-mail.
Any unauthorized copying, disclosure or distribution of the material
in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures and to
http://www.db.com/unitedkingdom/content/privacy.htm for information
about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev


Tom Doust
 

Hi Glenn

Are you saying that if a message doesn't carry the mdMsgType field then the OpenMAMA layer just passes it through to the client application. I hadn’t actually realised that ! I sort of assumed it was required and there would be some kind of error raised if it was missing.

Tom

-----Original Message-----
From: Glenn McClements [mailto:g.mcclements@srtechlabs.com]
Sent: 16 April 2015 14:29
To: Tom Doust; Alireza Assadzadeh; Yury Batrakov; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

Hi Tom,
I’m not sure what distinguishes the messages you suggest from any other kind of application messages, which don’t even need to have a message type. The ANK/NAK messages you suggest look like they’re dealt with at a custom block at the application level, so OpenMAMA just needs to be dumb in this respect and pass those messages though.

In summary the stack is the below:

1) Middleware
------------
2) Bridge

------------
3) OpenMAMA
-------------
4) Application



(Hopefully the above comes out the mail server formatted OK)

And the messages we’ve been talking about terminate at the levels:

1) Middleware heartbeats (generally) other internal middleware messages

2) Not too much currently terminates at this level, some middlewares pass back info messages or callbacks which may or may not be propagated up to OpenMAMA. Yesterday when I mentioned the possibility of adding a new INTERNAL message type for the use case mentioned, I was considering it terminating here, or possibly up to OpenMAMA which would ignore it.

3) Some OpenMAMA internal messages, such as REFRESHs.

4) Everything else, including NULL messages. Sorry Alireza, I missed your mail first time round

Heartbeats could theoretically live in any level, but For Yury’s use case it’s specifically 1 & 2, and therefore I don’t think these messages should be propagated up. NULL messages could be used, but then you would be adding expected behaviour onto the application level e.g. they should be ignored, and this is something I try to avoid.

I think there is room here to make this more explicit, possibly though a couple of new message types or namespaces but this could be complicating things.

Yury - it actually just occurred to me, are these heartbeats on a per transport, publisher or topic level? If they’re of the transport or publisher, then another possibility would be to have a separate middleware level subscription which you could send whatever you wanted over, and OpenMAMA wouldn’t be affected at all.

Regards,
Glenn





On 16/04/2015 10:21, "Tom Doust" <tom.doust@tick42.com> wrote:

Although I can think of many reasons why this would be a bad idea, it might be worth considering defining some MAMA_MSG_TYPE_PLATFORM

This would allow bridge developers to deliver custom stuff to client code with the intention of formalising into the api when it turns out to become useful.

An example of this would be the ACK/NAK messages our RMDS bridge delivers in response to message posts on TREP. We implemented these in the absence of any publisher response mechanism in OpenMAMA and when the new publisher callbacks that are currently under development appear in a version of OpenMAMA we will update the bridge to use those instead


Tom

-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Alireza Assadzadeh
Sent: 14 April 2015 17:06
To: Yury Batrakov; Glenn McClements; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

BTW, I noticed the following description for MAMA_MSG_TYPE_NULL in mamaMsgType. I did not see this message type having special logic associated with it in OpenMAMA MAMC/MAMACPP.

From msgtype.h:

/** Keep alive message. */
MAMA_MSG_TYPE_NULL = 175,

Is this message type suitable for this case?

--Alireza

-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-
dev-bounces@lists.openmama.org] On Behalf Of Yury Batrakov
Sent: Tuesday, April 14, 2015 11:03 AM
To: Glenn McClements; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Thanks Glenn,

BTW what is best practice of implementing custom messages? I mean what
is to return for mamaMsg_get(MdMsgType, 1), any integer which is not
in standard list of update types?


-----Original Message-----
From: Glenn McClements [mailto:g.mcclements@srtechlabs.com]
Sent: Tuesday, April 14, 2015 4:53 PM
To: Ian Bell; Yury Batrakov; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Thanks Ian, you beat me to it!

First of all Yury, welcome to the list and thanks for your question.

As Ian mentions one of the issues here is that you are implementing a
middleware level mechanism using OpenMAMA messages, so the solution is
not to modify dqpublishermanager to handle the new use of REFRESHs,
but to make sure that whatever messages are used for heartbeats are
never passed up to the core OpenMAMA level.


I’d also be against adding a new HEARTBEAT OpenMAMA message type for
this use case as would imply that heartbeats would be
sent/received/processed the OpenMAMA level (which is something we may
add at some point) and *not* at the middleware level. That’s not to
say you can’t add a new specific message type for your own heartbeats,
but it should be middleware/bridge specific.

Also a minor point is that mamaDQPublisher_sendReply() to me implies
replying to an mamaInbox, and for refreshes there is (currently) no
inbox to reply to.

Thanks,
Glenn

On 14/04/2015 14:34, "Ian Bell" <ian.bell@hapsoftconsulting.co.uk> wrote:

Hi Yury

The MAMA level refreshes are generally separate from the middleware
heartbeat functionality. Specially they operate at a higher level
although they can be similar in nature.

Normal operation would be to have the transport clean up the client
context information as the clients went away using its own heartbeat
functionality and let MAMA inform the client application of
subscribers using its own refresh messages. The refresh timeout is an
hour so this is generally a lot higher than a transport heartbeat timeout.

The dqpublisher specifically does a send rather than a sendReply to
reduce
the heartbeat traffic as well as to allow for transports which don't
maintain subscriber lists.

I.e. If you have 4 clients on 1 symbol each will send a refresh
message once
an hour less a random period of time, however the client refresh
timeout is reset on receiving a refresh response from a publisher, so
what actually happens is that the first client to timeout sends a
refresh and then the rest of the clients reset their timeouts on the refresh response from the publisher.
So the publisher only processes a single refresh message per topic as
opposed to 1 for every client on every topic.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk



-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org
[mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury
Batrakov
Sent: 14 April 2015 13:43
To: openmama-dev@lists.openmama.org
Subject: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Hi team,

I'm implementing new OpenMAMA transport bridge and have a question
about Advanced publishing. In my middleware a publisher keeps track of
all subscribers connected to it. A publisher and subscribers exchange
with heartbeat messages posted to a single topic providing a string
with session id in heartbeats' payload. When a publisher receives a
heartbeat it sends a reply with the same session ID. Once heartbeat is
not received by publisher it stops sending data and cleans up the resources allocated for a subscriber.
I tried implementing heartbeats with MAMA_SUBSC_REFRESH messages
but
the following problem occurs: in dqpublishermanager.c there is a
handler for REFRESHES

switch (msgType)
{
case MAMA_SUBSC_REFRESH:
if (!impl->mRefreshResponseMsg)
{
*Here MAMA creates empty reply message*
mamaMsg_create(&impl->mRefreshResponseMsg);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_MISC);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgType.mFid,
MAMA_MSG_TYPE_REFRESH);

}
*And sends it via the bridge*
mamaDQPublisher_send(info->pub, impl-
mRefreshResponseMsg);
impl-
mUserCallbacks.onRefresh((mamaDQPublisherManager)impl,
info, subType,
msgType, msg );


The problem is that in the bridge we cannot restore the context of
communication i.e. session id as mRefreshResponseMsg doesn't have any
fields set. Thus we force users to implement a part of underlying
protocol by handling refreshes themselves.

I'd propose the following change to fix that:

- mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
+ mamaDQPublisher_sendReply(info->pub, msg, impl-
mRefreshResponseMsg);

Or could you advise how to workaround it?

Thanks,
Yury.


---
This e-mail may contain confidential and/or privileged information.
If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and delete this e-mail. Any
unauthorized copying, disclosure or distribution of the material in
this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures and to
http://www.db.com/unitedkingdom/content/privacy.htm for information
about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev

---
This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient (or have received this e-mail in
error) please notify the sender immediately and delete this e-mail.
Any unauthorized copying, disclosure or distribution of the material
in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures and to
http://www.db.com/unitedkingdom/content/privacy.htm for information
about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev


Ian Bell
 

Hi

Just to clairify.

If it’s a marketdata subscription (ie MamaSubscription) then STATUS and TYPE are required fields. If they are not in the message the message will be dropped.

For basic subscriptions no fields are mandatory and everything passes through.

Thanks
Ian

-----Original Message-----
From: Tom Doust [mailto:tom.doust@tick42.com]
Sent: 16 April 2015 15:08
To: Glenn McClements; Alireza Assadzadeh; Yury Batrakov; Ian Bell; openmama-dev@lists.openmama.org
Subject: RE: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

Hi Glenn

Are you saying that if a message doesn't carry the mdMsgType field then the OpenMAMA layer just passes it through to the client application. I hadn’t actually realised that ! I sort of assumed it was required and there would be some kind of error raised if it was missing.

Tom


-----Original Message-----
From: Glenn McClements [mailto:g.mcclements@srtechlabs.com]
Sent: 16 April 2015 14:29
To: Tom Doust; Alireza Assadzadeh; Yury Batrakov; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

Hi Tom,
I’m not sure what distinguishes the messages you suggest from any other kind of application messages, which don’t even need to have a message type. The ANK/NAK messages you suggest look like they’re dealt with at a custom block at the application level, so OpenMAMA just needs to be dumb in this respect and pass those messages though.

In summary the stack is the below:

1) Middleware
------------
2) Bridge

------------
3) OpenMAMA
-------------
4) Application



(Hopefully the above comes out the mail server formatted OK)

And the messages we’ve been talking about terminate at the levels:

1) Middleware heartbeats (generally) other internal middleware messages

2) Not too much currently terminates at this level, some middlewares pass back info messages or callbacks which may or may not be propagated up to OpenMAMA. Yesterday when I mentioned the possibility of adding a new INTERNAL message type for the use case mentioned, I was considering it terminating here, or possibly up to OpenMAMA which would ignore it.

3) Some OpenMAMA internal messages, such as REFRESHs.

4) Everything else, including NULL messages. Sorry Alireza, I missed your mail first time round

Heartbeats could theoretically live in any level, but For Yury’s use case it’s specifically 1 & 2, and therefore I don’t think these messages should be propagated up. NULL messages could be used, but then you would be adding expected behaviour onto the application level e.g. they should be ignored, and this is something I try to avoid.

I think there is room here to make this more explicit, possibly though a couple of new message types or namespaces but this could be complicating things.

Yury - it actually just occurred to me, are these heartbeats on a per transport, publisher or topic level? If they’re of the transport or publisher, then another possibility would be to have a separate middleware level subscription which you could send whatever you wanted over, and OpenMAMA wouldn’t be affected at all.

Regards,
Glenn





On 16/04/2015 10:21, "Tom Doust" <tom.doust@tick42.com> wrote:

Although I can think of many reasons why this would be a bad idea, it might be worth considering defining some MAMA_MSG_TYPE_PLATFORM

This would allow bridge developers to deliver custom stuff to client code with the intention of formalising into the api when it turns out to become useful.

An example of this would be the ACK/NAK messages our RMDS bridge delivers in response to message posts on TREP. We implemented these in the absence of any publisher response mechanism in OpenMAMA and when the new publisher callbacks that are currently under development appear in a version of OpenMAMA we will update the bridge to use those instead


Tom

-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Alireza Assadzadeh
Sent: 14 April 2015 17:06
To: Yury Batrakov; Glenn McClements; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

BTW, I noticed the following description for MAMA_MSG_TYPE_NULL in mamaMsgType. I did not see this message type having special logic associated with it in OpenMAMA MAMC/MAMACPP.

From msgtype.h:

/** Keep alive message. */
MAMA_MSG_TYPE_NULL = 175,

Is this message type suitable for this case?

--Alireza

-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-
dev-bounces@lists.openmama.org] On Behalf Of Yury Batrakov
Sent: Tuesday, April 14, 2015 11:03 AM
To: Glenn McClements; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Thanks Glenn,

BTW what is best practice of implementing custom messages? I mean what
is to return for mamaMsg_get(MdMsgType, 1), any integer which is not
in standard list of update types?


-----Original Message-----
From: Glenn McClements [mailto:g.mcclements@srtechlabs.com]
Sent: Tuesday, April 14, 2015 4:53 PM
To: Ian Bell; Yury Batrakov; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Thanks Ian, you beat me to it!

First of all Yury, welcome to the list and thanks for your question.

As Ian mentions one of the issues here is that you are implementing a
middleware level mechanism using OpenMAMA messages, so the solution is
not to modify dqpublishermanager to handle the new use of REFRESHs,
but to make sure that whatever messages are used for heartbeats are
never passed up to the core OpenMAMA level.


I’d also be against adding a new HEARTBEAT OpenMAMA message type for
this use case as would imply that heartbeats would be
sent/received/processed the OpenMAMA level (which is something we may
add at some point) and *not* at the middleware level. That’s not to
say you can’t add a new specific message type for your own heartbeats,
but it should be middleware/bridge specific.

Also a minor point is that mamaDQPublisher_sendReply() to me implies
replying to an mamaInbox, and for refreshes there is (currently) no
inbox to reply to.

Thanks,
Glenn

On 14/04/2015 14:34, "Ian Bell" <ian.bell@hapsoftconsulting.co.uk> wrote:

Hi Yury

The MAMA level refreshes are generally separate from the middleware
heartbeat functionality. Specially they operate at a higher level
although they can be similar in nature.

Normal operation would be to have the transport clean up the client
context information as the clients went away using its own heartbeat
functionality and let MAMA inform the client application of
subscribers using its own refresh messages. The refresh timeout is an
hour so this is generally a lot higher than a transport heartbeat timeout.

The dqpublisher specifically does a send rather than a sendReply to
reduce
the heartbeat traffic as well as to allow for transports which don't
maintain subscriber lists.

I.e. If you have 4 clients on 1 symbol each will send a refresh
message once
an hour less a random period of time, however the client refresh
timeout is reset on receiving a refresh response from a publisher, so
what actually happens is that the first client to timeout sends a
refresh and then the rest of the clients reset their timeouts on the refresh response from the publisher.
So the publisher only processes a single refresh message per topic as
opposed to 1 for every client on every topic.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk



-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org
[mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury
Batrakov
Sent: 14 April 2015 13:43
To: openmama-dev@lists.openmama.org
Subject: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Hi team,

I'm implementing new OpenMAMA transport bridge and have a question
about Advanced publishing. In my middleware a publisher keeps track of
all subscribers connected to it. A publisher and subscribers exchange
with heartbeat messages posted to a single topic providing a string
with session id in heartbeats' payload. When a publisher receives a
heartbeat it sends a reply with the same session ID. Once heartbeat is
not received by publisher it stops sending data and cleans up the resources allocated for a subscriber.
I tried implementing heartbeats with MAMA_SUBSC_REFRESH messages
but
the following problem occurs: in dqpublishermanager.c there is a
handler for REFRESHES

switch (msgType)
{
case MAMA_SUBSC_REFRESH:
if (!impl->mRefreshResponseMsg)
{
*Here MAMA creates empty reply message*
mamaMsg_create(&impl->mRefreshResponseMsg);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_MISC);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgType.mFid,
MAMA_MSG_TYPE_REFRESH);

}
*And sends it via the bridge*
mamaDQPublisher_send(info->pub, impl-
mRefreshResponseMsg);
impl-
mUserCallbacks.onRefresh((mamaDQPublisherManager)impl,
info, subType,
msgType, msg );


The problem is that in the bridge we cannot restore the context of
communication i.e. session id as mRefreshResponseMsg doesn't have any
fields set. Thus we force users to implement a part of underlying
protocol by handling refreshes themselves.

I'd propose the following change to fix that:

- mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
+ mamaDQPublisher_sendReply(info->pub, msg, impl-
mRefreshResponseMsg);

Or could you advise how to workaround it?

Thanks,
Yury.


---
This e-mail may contain confidential and/or privileged information.
If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and delete this e-mail. Any
unauthorized copying, disclosure or distribution of the material in
this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures and to
http://www.db.com/unitedkingdom/content/privacy.htm for information
about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev

---
This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient (or have received this e-mail in
error) please notify the sender immediately and delete this e-mail.
Any unauthorized copying, disclosure or distribution of the material
in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures and to
http://www.db.com/unitedkingdom/content/privacy.htm for information
about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev


Glenn McClements <g.mcclements@...>
 

Yeah, as Ian points out, this entirely depends on whether it’s basic or market data subscriptions. For heartbeats I was originally assuming basic, but it’ll depend on the answer to my question in the previous mail. i.e. are the heartbeats

- at a transport or source level, which would imply basic subscription or a native middleware level subscription
- at the topic level, which would imply they are *part* of a market data subscription.

Glenn

On 16/04/2015 15:15, "Ian Bell" <ian.bell@hapsoftconsulting.co.uk> wrote:

Hi

Just to clairify.

If it’s a marketdata subscription (ie MamaSubscription) then STATUS and TYPE are required fields. If they are not in the message the message will be dropped.

For basic subscriptions no fields are mandatory and everything passes through.

Thanks
Ian

-----Original Message-----
From: Tom Doust [mailto:tom.doust@tick42.com]
Sent: 16 April 2015 15:08
To: Glenn McClements; Alireza Assadzadeh; Yury Batrakov; Ian Bell; openmama-dev@lists.openmama.org
Subject: RE: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

Hi Glenn

Are you saying that if a message doesn't carry the mdMsgType field then the OpenMAMA layer just passes it through to the client application. I hadn’t actually realised that ! I sort of assumed it was required and there would be some kind of error raised if it was missing.

Tom


-----Original Message-----
From: Glenn McClements [mailto:g.mcclements@srtechlabs.com]
Sent: 16 April 2015 14:29
To: Tom Doust; Alireza Assadzadeh; Yury Batrakov; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

Hi Tom,
I’m not sure what distinguishes the messages you suggest from any other kind of application messages, which don’t even need to have a message type. The ANK/NAK messages you suggest look like they’re dealt with at a custom block at the application level, so OpenMAMA just needs to be dumb in this respect and pass those messages though.

In summary the stack is the below:

1) Middleware
------------
2) Bridge

------------
3) OpenMAMA
-------------
4) Application



(Hopefully the above comes out the mail server formatted OK)

And the messages we’ve been talking about terminate at the levels:

1) Middleware heartbeats (generally) other internal middleware messages

2) Not too much currently terminates at this level, some middlewares pass back info messages or callbacks which may or may not be propagated up to OpenMAMA. Yesterday when I mentioned the possibility of adding a new INTERNAL message type for the use case mentioned, I was considering it terminating here, or possibly up to OpenMAMA which would ignore it.

3) Some OpenMAMA internal messages, such as REFRESHs.

4) Everything else, including NULL messages. Sorry Alireza, I missed your mail first time round

Heartbeats could theoretically live in any level, but For Yury’s use case it’s specifically 1 & 2, and therefore I don’t think these messages should be propagated up. NULL messages could be used, but then you would be adding expected behaviour onto the application level e.g. they should be ignored, and this is something I try to avoid.

I think there is room here to make this more explicit, possibly though a couple of new message types or namespaces but this could be complicating things.

Yury - it actually just occurred to me, are these heartbeats on a per transport, publisher or topic level? If they’re of the transport or publisher, then another possibility would be to have a separate middleware level subscription which you could send whatever you wanted over, and OpenMAMA wouldn’t be affected at all.

Regards,
Glenn





On 16/04/2015 10:21, "Tom Doust" <tom.doust@tick42.com> wrote:

Although I can think of many reasons why this would be a bad idea, it might be worth considering defining some MAMA_MSG_TYPE_PLATFORM

This would allow bridge developers to deliver custom stuff to client code with the intention of formalising into the api when it turns out to become useful.

An example of this would be the ACK/NAK messages our RMDS bridge delivers in response to message posts on TREP. We implemented these in the absence of any publisher response mechanism in OpenMAMA and when the new publisher callbacks that are currently under development appear in a version of OpenMAMA we will update the bridge to use those instead


Tom

-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Alireza Assadzadeh
Sent: 14 April 2015 17:06
To: Yury Batrakov; Glenn McClements; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher manager

BTW, I noticed the following description for MAMA_MSG_TYPE_NULL in mamaMsgType. I did not see this message type having special logic associated with it in OpenMAMA MAMC/MAMACPP.

From msgtype.h:

/** Keep alive message. */
MAMA_MSG_TYPE_NULL = 175,

Is this message type suitable for this case?

--Alireza

-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-
dev-bounces@lists.openmama.org] On Behalf Of Yury Batrakov
Sent: Tuesday, April 14, 2015 11:03 AM
To: Glenn McClements; Ian Bell; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Thanks Glenn,

BTW what is best practice of implementing custom messages? I mean what
is to return for mamaMsg_get(MdMsgType, 1), any integer which is not
in standard list of update types?


-----Original Message-----
From: Glenn McClements [mailto:g.mcclements@srtechlabs.com]
Sent: Tuesday, April 14, 2015 4:53 PM
To: Ian Bell; Yury Batrakov; openmama-dev@lists.openmama.org
Subject: Re: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Thanks Ian, you beat me to it!

First of all Yury, welcome to the list and thanks for your question.

As Ian mentions one of the issues here is that you are implementing a
middleware level mechanism using OpenMAMA messages, so the solution is
not to modify dqpublishermanager to handle the new use of REFRESHs,
but to make sure that whatever messages are used for heartbeats are
never passed up to the core OpenMAMA level.


I’d also be against adding a new HEARTBEAT OpenMAMA message type for
this use case as would imply that heartbeats would be
sent/received/processed the OpenMAMA level (which is something we may
add at some point) and *not* at the middleware level. That’s not to
say you can’t add a new specific message type for your own heartbeats,
but it should be middleware/bridge specific.

Also a minor point is that mamaDQPublisher_sendReply() to me implies
replying to an mamaInbox, and for refreshes there is (currently) no
inbox to reply to.

Thanks,
Glenn

On 14/04/2015 14:34, "Ian Bell" <ian.bell@hapsoftconsulting.co.uk> wrote:

Hi Yury

The MAMA level refreshes are generally separate from the middleware
heartbeat functionality. Specially they operate at a higher level
although they can be similar in nature.

Normal operation would be to have the transport clean up the client
context information as the clients went away using its own heartbeat
functionality and let MAMA inform the client application of
subscribers using its own refresh messages. The refresh timeout is an
hour so this is generally a lot higher than a transport heartbeat timeout.

The dqpublisher specifically does a send rather than a sendReply to
reduce
the heartbeat traffic as well as to allow for transports which don't
maintain subscriber lists.

I.e. If you have 4 clients on 1 symbol each will send a refresh
message once
an hour less a random period of time, however the client refresh
timeout is reset on receiving a refresh response from a publisher, so
what actually happens is that the first client to timeout sends a
refresh and then the rest of the clients reset their timeouts on the refresh response from the publisher.
So the publisher only processes a single refresh message per topic as
opposed to 1 for every client on every topic.

Thanks,
Ian Bell

Hapsoft Consulting Ltd
138 Belmont Road, Belfast, BT4 2AQ
Office: +447481866051
Web: www.hapsoftconsulting.co.uk



-----Original Message-----
From: openmama-dev-bounces@lists.openmama.org
[mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Yury
Batrakov
Sent: 14 April 2015 13:43
To: openmama-dev@lists.openmama.org
Subject: [Openmama-dev] Handling REFRESH messages in DQ publisher
manager

Classification: Public

Hi team,

I'm implementing new OpenMAMA transport bridge and have a question
about Advanced publishing. In my middleware a publisher keeps track of
all subscribers connected to it. A publisher and subscribers exchange
with heartbeat messages posted to a single topic providing a string
with session id in heartbeats' payload. When a publisher receives a
heartbeat it sends a reply with the same session ID. Once heartbeat is
not received by publisher it stops sending data and cleans up the resources allocated for a subscriber.
I tried implementing heartbeats with MAMA_SUBSC_REFRESH messages
but
the following problem occurs: in dqpublishermanager.c there is a
handler for REFRESHES

switch (msgType)
{
case MAMA_SUBSC_REFRESH:
if (!impl->mRefreshResponseMsg)
{
*Here MAMA creates empty reply message*
mamaMsg_create(&impl->mRefreshResponseMsg);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_MISC);
mamaMsg_addU8(impl->mRefreshResponseMsg, NULL,
MamaFieldMsgType.mFid,
MAMA_MSG_TYPE_REFRESH);

}
*And sends it via the bridge*
mamaDQPublisher_send(info->pub, impl-
mRefreshResponseMsg);
impl-
mUserCallbacks.onRefresh((mamaDQPublisherManager)impl,
info, subType,
msgType, msg );


The problem is that in the bridge we cannot restore the context of
communication i.e. session id as mRefreshResponseMsg doesn't have any
fields set. Thus we force users to implement a part of underlying
protocol by handling refreshes themselves.

I'd propose the following change to fix that:

- mamaDQPublisher_send(info->pub, impl->mRefreshResponseMsg);
+ mamaDQPublisher_sendReply(info->pub, msg, impl-
mRefreshResponseMsg);

Or could you advise how to workaround it?

Thanks,
Yury.


---
This e-mail may contain confidential and/or privileged information.
If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and delete this e-mail. Any
unauthorized copying, disclosure or distribution of the material in
this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures and to
http://www.db.com/unitedkingdom/content/privacy.htm for information
about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev

---
This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient (or have received this e-mail in
error) please notify the sender immediately and delete this e-mail.
Any unauthorized copying, disclosure or distribution of the material
in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures and to
http://www.db.com/unitedkingdom/content/privacy.htm for information
about privacy.
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev
_______________________________________________
Openmama-dev mailing list
Openmama-dev@lists.openmama.org
https://lists.openmama.org/mailman/listinfo/openmama-dev