Topics

New feature: Extending timestamp format


Stuart Beattie
 

Hi everyone,

 

We are currently investigating an approach to improve and extend the handling of date/times in OpenMAMA.

 

The current mamaDateTime type, essentially being a 64-bit bitmask, is inflexible and has some built in limitations – such as the 4-byte second field limiting the maximum stored date to the year 2106, as well as being limited to time since the epoch.  The current datetime format is this:

Data Content

Size

Comments

Seconds

32 bits

Needs to be bigger to support times beyond 2106 and also has no ability to represent times before 1970.

Microseconds

20 bits

Maximum value is 1,048,575 - if nanosecond support is ever required, this would be too small (you'd need another 10 bits)

Precision

4 bits

Used to store decimal precision.

Hints

4 bits

Used to store information on whether or not to display as date / time (one of these bits going spare too, could expand to 2 with changes)

Spare (idle)

4 bits

Could be 6 if you count the values spare in hints

 

So while there are currently some bytes available to extend this somewhat, if these are used there will be no more room for expansion or new features in the existing format so we are thinking of a more long-term solution.

 

Instead we are proposing to introduce a new datetime format using a new type which is an actual struct with separate fields for the different elements (seconds, date, time etc.)  While this may be larger (in terms of storage size – eg some bits may go unused), it would have the advantages of:

a)      Being more extensible – fields could be added or changed as necessary without being bound by the current 64 bits

b)      Being more intuitive to work with

 

Format could be something like:

 

Data Content

Data Type

Comments

Seconds

int64 (yes - signed)

Not every bit will be used, but easy to work with and will support times prior to 1970.

Nanoseconds?

long int

(Current format has only microseconds)

Precision

uint8

Again, some bits going spare, but easy to work with.

Hints

uint8

Again, some bits going spare, but easy to work with.

 

 

This is in the early stages so we are asking for input as well as having a few specific questions for developers – with backwards compatibility being a major concern:

-          It is currently possible to serialise a mamaDateTime by using the native bitmask format directly – is anyone out there currently doing this and relying on this being possible?

-          For bridge developers, how do you currently encode timestamps?

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

Thanks

Stuart

 

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC


Phelan, Nigel
 

If contemplating a new type, is it worth considering the question of time zones?  The current model doesn’t really address this, although the java constructors have some implicit assumptions that DateTime objects are stored in UTC

 

Nigel

 


Nigel Phelan | Corporate & Investment Bank | Market Data Services | J.P. Morgan

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Stuart Beattie
Sent: Wednesday, September 28, 2016 5:23 PM
To: openmama-dev@...
Subject: [Openmama-dev] New feature: Extending timestamp format

 

Hi everyone,

 

We are currently investigating an approach to improve and extend the handling of date/times in OpenMAMA.

 

The current mamaDateTime type, essentially being a 64-bit bitmask, is inflexible and has some built in limitations – such as the 4-byte second field limiting the maximum stored date to the year 2106, as well as being limited to time since the epoch.  The current datetime format is this:

 

Data Content

Size

Comments

Seconds

32 bits

Needs to be bigger to support times beyond 2106 and also has no ability to represent times before 1970.

Microseconds

20 bits

Maximum value is 1,048,575 - if nanosecond support is ever required, this would be too small (you'd need another 10 bits)

Precision

4 bits

Used to store decimal precision.

Hints

4 bits

Used to store information on whether or not to display as date / time (one of these bits going spare too, could expand to 2 with changes)

Spare (idle)

4 bits

Could be 6 if you count the values spare in hints

 

So while there are currently some bytes available to extend this somewhat, if these are used there will be no more room for expansion or new features in the existing format so we are thinking of a more long-term solution.

 

Instead we are proposing to introduce a new datetime format using a new type which is an actual struct with separate fields for the different elements (seconds, date, time etc.)  While this may be larger (in terms of storage size – eg some bits may go unused), it would have the advantages of:

a)      Being more extensible – fields could be added or changed as necessary without being bound by the current 64 bits

b)      Being more intuitive to work with

 

Format could be something like:

 

Data Content

Data Type

Comments

Seconds

int64 (yes - signed)

Not every bit will be used, but easy to work with and will support times prior to 1970.

Nanoseconds?

long int

(Current format has only microseconds)

Precision

uint8

Again, some bits going spare, but easy to work with.

Hints

uint8

Again, some bits going spare, but easy to work with.

 

 

This is in the early stages so we are asking for input as well as having a few specific questions for developers – with backwards compatibility being a major concern:

-          It is currently possible to serialise a mamaDateTime by using the native bitmask format directly – is anyone out there currently doing this and relying on this being possible?

-          For bridge developers, how do you currently encode timestamps?

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

Thanks

Stuart

 

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email


Stuart Beattie
 

Hi Nigel,

 

Apologies, yes I meant to mention timezones in the original email as something to consider

 

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

“addition of a timestamp” was meant to be “timezone”, my mistake.

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

From: Phelan, Nigel [mailto:nigel.phelan@...]
Sent: 28 September 2016 17:36
To: Stuart Beattie <sbeattie@...>; openmama-dev@...
Subject: RE: New feature: Extending timestamp format

 

If contemplating a new type, is it worth considering the question of time zones?  The current model doesn’t really address this, although the java constructors have some implicit assumptions that DateTime objects are stored in UTC

 

Nigel

 


Nigel Phelan | Corporate & Investment Bank | Market Data Services | J.P. Morgan

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Stuart Beattie
Sent: Wednesday, September 28, 2016 5:23 PM
To: openmama-dev@...
Subject: [Openmama-dev] New feature: Extending timestamp format

 

Hi everyone,

 

We are currently investigating an approach to improve and extend the handling of date/times in OpenMAMA.

 

The current mamaDateTime type, essentially being a 64-bit bitmask, is inflexible and has some built in limitations – such as the 4-byte second field limiting the maximum stored date to the year 2106, as well as being limited to time since the epoch.  The current datetime format is this:

 

Data Content

Size

Comments

Seconds

32 bits

Needs to be bigger to support times beyond 2106 and also has no ability to represent times before 1970.

Microseconds

20 bits

Maximum value is 1,048,575 - if nanosecond support is ever required, this would be too small (you'd need another 10 bits)

Precision

4 bits

Used to store decimal precision.

Hints

4 bits

Used to store information on whether or not to display as date / time (one of these bits going spare too, could expand to 2 with changes)

Spare (idle)

4 bits

Could be 6 if you count the values spare in hints

 

So while there are currently some bytes available to extend this somewhat, if these are used there will be no more room for expansion or new features in the existing format so we are thinking of a more long-term solution.

 

Instead we are proposing to introduce a new datetime format using a new type which is an actual struct with separate fields for the different elements (seconds, date, time etc.)  While this may be larger (in terms of storage size – eg some bits may go unused), it would have the advantages of:

a)      Being more extensible – fields could be added or changed as necessary without being bound by the current 64 bits

b)      Being more intuitive to work with

 

Format could be something like:

 

Data Content

Data Type

Comments

Seconds

int64 (yes - signed)

Not every bit will be used, but easy to work with and will support times prior to 1970.

Nanoseconds?

long int

(Current format has only microseconds)

Precision

uint8

Again, some bits going spare, but easy to work with.

Hints

uint8

Again, some bits going spare, but easy to work with.

 

 

This is in the early stages so we are asking for input as well as having a few specific questions for developers – with backwards compatibility being a major concern:

-          It is currently possible to serialise a mamaDateTime by using the native bitmask format directly – is anyone out there currently doing this and relying on this being possible?

-          For bridge developers, how do you currently encode timestamps?

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

Thanks

Stuart

 

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

 


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC


Phelan, Nigel
 

It’s definitely a good thing to have, Glenn, but it does add some complexity, particularly when you consider future and past dates, daylight savings transitions and the desire to compare timestamps for latency measurement purposes.  I think we should try to keep the scope manageable, but we are finding the date ranges restrictive with real world use cases like mortgage backed securities and long dated bonds

 

Nigel

 


Nigel Phelan | Corporate & Investment Bank | Market Data Services | J.P. Morgan

 

From: Glenn McClements [mailto:gmcclements@...]
Sent: Thursday, September 29, 2016 4:38 AM
To: Stuart Beattie; Phelan, Nigel; openmama-dev@...
Subject: Re: New feature: Extending timestamp format

 

>If contemplating a new type, is it worth considering the question of time zones?

 

The current OpenMAMA implementation assumes all timestamps are UTC, and as much as I like the simplicity of this, even in small deployments it can prove difficult to guarantee even in small deployments. So I do think some timezone capabilities are needed in the longer term, and it's actually something which sprang to mind as well when were were considering these changes. 

 

A question would be where OpenMAMA picks up the timezone from. It could be defined in a MamaSource metadata, reference (initial) data for the symbol, some other out of band mechanism or encoded in the field, but I think this a separate problem from supporting it in the MamaDateTime type.

 

Glenn 

 

 

GLENN MCCLEMENTS

SVP Engineering, Europe

 

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 


From: openmama-dev-bounces@... <openmama-dev-bounces@...> on behalf of Stuart Beattie <sbeattie@...>
Sent: 28 September 2016 17:39
To: Phelan, Nigel; openmama-dev@...
Subject: Re: [Openmama-dev] New feature: Extending timestamp format

 

Hi Nigel,

 

Apologies, yes I meant to mention timezones in the original email as something to consider

 

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

“addition of a timestamp” was meant to be “timezone”, my mistake.

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

From: Phelan, Nigel [mailto:nigel.phelan@...]
Sent: 28 September 2016 17:36
To: Stuart Beattie <sbeattie@...>; openmama-dev@...
Subject: RE: New feature: Extending timestamp format

 

If contemplating a new type, is it worth considering the question of time zones?  The current model doesn’t really address this, although the java constructors have some implicit assumptions that DateTime objects are stored in UTC

 

Nigel

 


Nigel Phelan | Corporate & Investment Bank | Market Data Services | J.P. Morgan

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Stuart Beattie
Sent: Wednesday, September 28, 2016 5:23 PM
To: openmama-dev@...
Subject: [Openmama-dev] New feature: Extending timestamp format

 

Hi everyone,

 

We are currently investigating an approach to improve and extend the handling of date/times in OpenMAMA.

 

The current mamaDateTime type, essentially being a 64-bit bitmask, is inflexible and has some built in limitations – such as the 4-byte second field limiting the maximum stored date to the year 2106, as well as being limited to time since the epoch.  The current datetime format is this:

 

Data Content

Size

Comments

Seconds

32 bits

Needs to be bigger to support times beyond 2106 and also has no ability to represent times before 1970.

Microseconds

20 bits

Maximum value is 1,048,575 - if nanosecond support is ever required, this would be too small (you'd need another 10 bits)

Precision

4 bits

Used to store decimal precision.

Hints

4 bits

Used to store information on whether or not to display as date / time (one of these bits going spare too, could expand to 2 with changes)

Spare (idle)

4 bits

Could be 6 if you count the values spare in hints

 

So while there are currently some bytes available to extend this somewhat, if these are used there will be no more room for expansion or new features in the existing format so we are thinking of a more long-term solution.

 

Instead we are proposing to introduce a new datetime format using a new type which is an actual struct with separate fields for the different elements (seconds, date, time etc.)  While this may be larger (in terms of storage size – eg some bits may go unused), it would have the advantages of:

a)      Being more extensible – fields could be added or changed as necessary without being bound by the current 64 bits

b)      Being more intuitive to work with

 

Format could be something like:

 

Data Content

Data Type

Comments

Seconds

int64 (yes - signed)

Not every bit will be used, but easy to work with and will support times prior to 1970.

Nanoseconds?

long int

(Current format has only microseconds)

Precision

uint8

Again, some bits going spare, but easy to work with.

Hints

uint8

Again, some bits going spare, but easy to work with.

 

 

This is in the early stages so we are asking for input as well as having a few specific questions for developers – with backwards compatibility being a major concern:

-          It is currently possible to serialise a mamaDateTime by using the native bitmask format directly – is anyone out there currently doing this and relying on this being possible?

-          For bridge developers, how do you currently encode timestamps?

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

Thanks

Stuart

 

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

 


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email


Phelan, Nigel
 

I think that, at the moment, the most important payload bridges come from Vela, Solace and Tick 42.  Would anyone from Solace and Tick 42 care to comment on the behaviour of their payload bridges and whether they took this shortcut in the implementation (I’m assuming Frank can speak authoritatively for Vela)?  Also good to know the QPID open source reference implementation gets this right.  Out of curiosity, Frank, does your ZeroMQ bridge handle this, or does it just re-use the proton payload (which would presumably automatically make it compatible)?

 

Thanks

 

Nigel

 


Nigel Phelan | Corporate & Investment Bank | Market Data Services | J.P. Morgan

 

From: Frank Quinn [mailto:fquinn@...]
Sent: Tuesday, October 04, 2016 9:13 AM
To: Stuart Beattie; Phelan, Nigel; openmama-dev@...
Subject: RE: New feature: Extending timestamp format

 

Hi Folks,

 

The biggest concern here is around interface compatibility with the mamaDateTime type. In our types.h it’s defined as a u64* rather than a void*. Unfortunately, I expect this has given payload bridges a mechanism too tempting to ignore when it comes to simply using the underlying U64 for serialization and deserialization.

 

If payload bridge developers have treated the mamaDateTime as an encapsulated object, changing mamaDateTime to an opaque struct would be reasonably safe. Qpid proton, for example, would be unaffected by the struct change as it uses mamaDateTime_get/set.

 

However, any payload bridges which access the native U64 would need to change. Options would be either:

 

1.       Just break the bridge and move on to 6.2

2.       Make the first member of the struct the “legacy” U64 member and have a payload bridge flag to enable “backwards compatible” mode and speckle a bunch of backwards compatible code throughout the MAMA Date Time code

3.       Assume that no bridges actually use this U64 directly and risk breaking the bridge

 

Personally my vote would be for #1 - this sort of thing is the reason why we put that version handshaking code in.

 

Note I just looked through the OpenMAMA wiki for a link explaining the versioning and didn’t find one, so I just created https://github.com/OpenMAMA/OpenMAMA/wiki/OpenMAMA-Versioning for reference.

 

Cheers,

Frank

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Stuart Beattie
Sent: Wednesday, September 28, 2016 5:39 PM
To: Phelan, Nigel <nigel.phelan@...>; openmama-dev@...
Subject: Re: [Openmama-dev] New feature: Extending timestamp format

 

Hi Nigel,

 

Apologies, yes I meant to mention timezones in the original email as something to consider

 

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

“addition of a timestamp” was meant to be “timezone”, my mistake.

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

From: Phelan, Nigel [mailto:nigel.phelan@...]
Sent: 28 September 2016 17:36
To: Stuart Beattie <sbeattie@...>; openmama-dev@...
Subject: RE: New feature: Extending timestamp format

 

If contemplating a new type, is it worth considering the question of time zones?  The current model doesn’t really address this, although the java constructors have some implicit assumptions that DateTime objects are stored in UTC

 

Nigel

 


Nigel Phelan | Corporate & Investment Bank | Market Data Services | J.P. Morgan

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Stuart Beattie
Sent: Wednesday, September 28, 2016 5:23 PM
To: openmama-dev@...
Subject: [Openmama-dev] New feature: Extending timestamp format

 

Hi everyone,

 

We are currently investigating an approach to improve and extend the handling of date/times in OpenMAMA.

 

The current mamaDateTime type, essentially being a 64-bit bitmask, is inflexible and has some built in limitations – such as the 4-byte second field limiting the maximum stored date to the year 2106, as well as being limited to time since the epoch.  The current datetime format is this:

 

Data Content

Size

Comments

Seconds

32 bits

Needs to be bigger to support times beyond 2106 and also has no ability to represent times before 1970.

Microseconds

20 bits

Maximum value is 1,048,575 - if nanosecond support is ever required, this would be too small (you'd need another 10 bits)

Precision

4 bits

Used to store decimal precision.

Hints

4 bits

Used to store information on whether or not to display as date / time (one of these bits going spare too, could expand to 2 with changes)

Spare (idle)

4 bits

Could be 6 if you count the values spare in hints

 

So while there are currently some bytes available to extend this somewhat, if these are used there will be no more room for expansion or new features in the existing format so we are thinking of a more long-term solution.

 

Instead we are proposing to introduce a new datetime format using a new type which is an actual struct with separate fields for the different elements (seconds, date, time etc.)  While this may be larger (in terms of storage size – eg some bits may go unused), it would have the advantages of:

a)      Being more extensible – fields could be added or changed as necessary without being bound by the current 64 bits

b)      Being more intuitive to work with

 

Format could be something like:

 

Data Content

Data Type

Comments

Seconds

int64 (yes - signed)

Not every bit will be used, but easy to work with and will support times prior to 1970.

Nanoseconds?

long int

(Current format has only microseconds)

Precision

uint8

Again, some bits going spare, but easy to work with.

Hints

uint8

Again, some bits going spare, but easy to work with.

 

 

This is in the early stages so we are asking for input as well as having a few specific questions for developers – with backwards compatibility being a major concern:

-          It is currently possible to serialise a mamaDateTime by using the native bitmask format directly – is anyone out there currently doing this and relying on this being possible?

-          For bridge developers, how do you currently encode timestamps?

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

Thanks

Stuart

 

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

 


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email


Glenn McClements <gmcclements@...>
 

For Vela we have a couple of payloads:

- V5/Wirecache has its own internal format and calls mamaDateTime setters, so this is fine.

- WombatMsg actually uses the same encoding as OpenMAMA does internally, so this will need to be changed but I’m fine with this as it was somewhat breaking encapsulation. The wire format will remain the same, we’ll just need to call a setting rather than doing a direct assignment.  

 

 

Also, from another offline thread, we should take time to consider if the new implementation will be flexible/extendable. Time zone hint and nanosecond support are two feature that spring to mind that we may like to add later.

 

These are outside of scope for the current piece of work but they have been asked about before and we’d want to make sure that any implementation could be extended to support these if needed.

 

 

GLENN MCCLEMENTS

SVP Engineering, Europe

 

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

 

From: <openmama-dev-bounces@...> on behalf of "Phelan, Nigel via Openmama-dev" <openmama-dev@...>
Reply-To: "Phelan, Nigel" <nigel.phelan@...>
Date: Tuesday, 4 October 2016 at 09:38
To: Frank Quinn <fquinn@...>, Stuart Beattie <sbeattie@...>, "openmama-dev@..." <openmama-dev@...>
Subject: Re: [Openmama-dev] New feature: Extending timestamp format

 

I think that, at the moment, the most important payload bridges come from Vela, Solace and Tick 42.  Would anyone from Solace and Tick 42 care to comment on the behaviour of their payload bridges and whether they took this shortcut in the implementation (I’m assuming Frank can speak authoritatively for Vela)?  Also good to know the QPID open source reference implementation gets this right.  Out of curiosity, Frank, does your ZeroMQ bridge handle this, or does it just re-use the proton payload (which would presumably automatically make it compatible)?

 

Thanks

 

Nigel

 


Nigel Phelan | Corporate & Investment Bank | Market Data Services | J.P. Morgan

 

From: Frank Quinn [mailto:fquinn@...]
Sent: Tuesday, October 04, 2016 9:13 AM
To: Stuart Beattie; Phelan, Nigel; openmama-dev@...
Subject: RE: New feature: Extending timestamp format

 

Hi Folks,

 

The biggest concern here is around interface compatibility with the mamaDateTime type. In our types.h it’s defined as a u64* rather than a void*. Unfortunately, I expect this has given payload bridges a mechanism too tempting to ignore when it comes to simply using the underlying U64 for serialization and deserialization.

 

If payload bridge developers have treated the mamaDateTime as an encapsulated object, changing mamaDateTime to an opaque struct would be reasonably safe. Qpid proton, for example, would be unaffected by the struct change as it uses mamaDateTime_get/set.

 

However, any payload bridges which access the native U64 would need to change. Options would be either:

 

1.       Just break the bridge and move on to 6.2

2.       Make the first member of the struct the “legacy” U64 member and have a payload bridge flag to enable “backwards compatible” mode and speckle a bunch of backwards compatible code throughout the MAMA Date Time code

3.       Assume that no bridges actually use this U64 directly and risk breaking the bridge

 

Personally my vote would be for #1 - this sort of thing is the reason why we put that version handshaking code in.

 

Note I just looked through the OpenMAMA wiki for a link explaining the versioning and didn’t find one, so I just created https://github.com/OpenMAMA/OpenMAMA/wiki/OpenMAMA-Versioning for reference.

 

Cheers,

Frank

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Stuart Beattie
Sent: Wednesday, September 28, 2016 5:39 PM
To: Phelan, Nigel <nigel.phelan@...>; openmama-dev@...
Subject: Re: [Openmama-dev] New feature: Extending timestamp format

 

Hi Nigel,

 

Apologies, yes I meant to mention timezones in the original email as something to consider

 

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

“addition of a timestamp” was meant to be “timezone”, my mistake.

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

From: Phelan, Nigel [mailto:nigel.phelan@...]
Sent: 28 September 2016 17:36
To: Stuart Beattie <sbeattie@...>; openmama-dev@...
Subject: RE: New feature: Extending timestamp format

 

If contemplating a new type, is it worth considering the question of time zones?  The current model doesn’t really address this, although the java constructors have some implicit assumptions that DateTime objects are stored in UTC

 

Nigel

 


Nigel Phelan | Corporate & Investment Bank | Market Data Services | J.P. Morgan

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Stuart Beattie
Sent: Wednesday, September 28, 2016 5:23 PM
To: openmama-dev@...
Subject: [Openmama-dev] New feature: Extending timestamp format

 

Hi everyone,

 

We are currently investigating an approach to improve and extend the handling of date/times in OpenMAMA.

 

The current mamaDateTime type, essentially being a 64-bit bitmask, is inflexible and has some built in limitations – such as the 4-byte second field limiting the maximum stored date to the year 2106, as well as being limited to time since the epoch.  The current datetime format is this:

 

Data Content

Size

Comments

Seconds

32 bits

Needs to be bigger to support times beyond 2106 and also has no ability to represent times before 1970.

Microseconds

20 bits

Maximum value is 1,048,575 - if nanosecond support is ever required, this would be too small (you'd need another 10 bits)

Precision

4 bits

Used to store decimal precision.

Hints

4 bits

Used to store information on whether or not to display as date / time (one of these bits going spare too, could expand to 2 with changes)

Spare (idle)

4 bits

Could be 6 if you count the values spare in hints

 

So while there are currently some bytes available to extend this somewhat, if these are used there will be no more room for expansion or new features in the existing format so we are thinking of a more long-term solution.

 

Instead we are proposing to introduce a new datetime format using a new type which is an actual struct with separate fields for the different elements (seconds, date, time etc.)  While this may be larger (in terms of storage size – eg some bits may go unused), it would have the advantages of:

a)      Being more extensible – fields could be added or changed as necessary without being bound by the current 64 bits

b)      Being more intuitive to work with

 

Format could be something like:

 

Data Content

Data Type

Comments

Seconds

int64 (yes - signed)

Not every bit will be used, but easy to work with and will support times prior to 1970.

Nanoseconds?

long int

(Current format has only microseconds)

Precision

uint8

Again, some bits going spare, but easy to work with.

Hints

uint8

Again, some bits going spare, but easy to work with.

 

 

This is in the early stages so we are asking for input as well as having a few specific questions for developers – with backwards compatibility being a major concern:

-          It is currently possible to serialise a mamaDateTime by using the native bitmask format directly – is anyone out there currently doing this and relying on this being possible?

-          For bridge developers, how do you currently encode timestamps?

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

Thanks

Stuart

 

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

 


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC


Tom Doust
 

In the TREP bridge we have a wrapper around the mamaDateTime type and only use the accessor functions. There is a certain amount of messing around converting between mamaDateTime and rssl date and time representations, but so long as the set of accessor functions doesn’t change it should “just work”

 

One to bear in mind with “breaking changes” like this is that you may also introduce a dependency between the bridge version and the mama runtime version. This was (is) an issue with the change to bridge loading. If I simply install a bridge built with OpenMAMA  > 2.4 into a deployment that includes  components built with OpenMama <2.4

 

But, in principle, I agree with Frank’s #1

 

Tom

 

 

From: <openmama-dev-bounces@...> on behalf of Glenn McClements <gmcclements@...>
Date: Tuesday, 4 October 2016 at 13:39
To: Nigel Phelan <nigel.phelan@...>, Frank Quinn <fquinn@...>, Stuart Beattie <sbeattie@...>, "Openmama-dev@..." <openmama-dev@...>
Subject: Re: [Openmama-dev] New feature: Extending timestamp format

 

For Vela we have a couple of payloads:

- V5/Wirecache has its own internal format and calls mamaDateTime setters, so this is fine.

- WombatMsg actually uses the same encoding as OpenMAMA does internally, so this will need to be changed but I’m fine with this as it was somewhat breaking encapsulation. The wire format will remain the same, we’ll just need to call a setting rather than doing a direct assignment.  

 

 

Also, from another offline thread, we should take time to consider if the new implementation will be flexible/extendable. Time zone hint and nanosecond support are two feature that spring to mind that we may like to add later.

 

These are outside of scope for the current piece of work but they have been asked about before and we’d want to make sure that any implementation could be extended to support these if needed.

 

 

GLENN MCCLEMENTS

SVP Engineering, Europe

 

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

 

From: <openmama-dev-bounces@...> on behalf of "Phelan, Nigel via Openmama-dev" <openmama-dev@...>
Reply-To: "Phelan, Nigel" <nigel.phelan@...>
Date: Tuesday, 4 October 2016 at 09:38
To: Frank Quinn <fquinn@...>, Stuart Beattie <sbeattie@...>, "openmama-dev@..." <openmama-dev@...>
Subject: Re: [Openmama-dev] New feature: Extending timestamp format

 

I think that, at the moment, the most important payload bridges come from Vela, Solace and Tick 42.  Would anyone from Solace and Tick 42 care to comment on the behaviour of their payload bridges and whether they took this shortcut in the implementation (I’m assuming Frank can speak authoritatively for Vela)?  Also good to know the QPID open source reference implementation gets this right.  Out of curiosity, Frank, does your ZeroMQ bridge handle this, or does it just re-use the proton payload (which would presumably automatically make it compatible)?

 

Thanks

 

Nigel

 


Nigel Phelan | Corporate & Investment Bank | Market Data Services | J.P. Morgan

 

From: Frank Quinn [mailto:fquinn@...]
Sent: Tuesday, October 04, 2016 9:13 AM
To: Stuart Beattie; Phelan, Nigel; openmama-dev@...
Subject: RE: New feature: Extending timestamp format

 

Hi Folks,

 

The biggest concern here is around interface compatibility with the mamaDateTime type. In our types.h it’s defined as a u64* rather than a void*. Unfortunately, I expect this has given payload bridges a mechanism too tempting to ignore when it comes to simply using the underlying U64 for serialization and deserialization.

 

If payload bridge developers have treated the mamaDateTime as an encapsulated object, changing mamaDateTime to an opaque struct would be reasonably safe. Qpid proton, for example, would be unaffected by the struct change as it uses mamaDateTime_get/set.

 

However, any payload bridges which access the native U64 would need to change. Options would be either:

 

1.       Just break the bridge and move on to 6.2

2.       Make the first member of the struct the “legacy” U64 member and have a payload bridge flag to enable “backwards compatible” mode and speckle a bunch of backwards compatible code throughout the MAMA Date Time code

3.       Assume that no bridges actually use this U64 directly and risk breaking the bridge

 

Personally my vote would be for #1 - this sort of thing is the reason why we put that version handshaking code in.

 

Note I just looked through the OpenMAMA wiki for a link explaining the versioning and didn’t find one, so I just created https://github.com/OpenMAMA/OpenMAMA/wiki/OpenMAMA-Versioning for reference.

 

Cheers,

Frank

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Stuart Beattie
Sent: Wednesday, September 28, 2016 5:39 PM
To: Phelan, Nigel <nigel.phelan@...>; openmama-dev@...
Subject: Re: [Openmama-dev] New feature: Extending timestamp format

 

Hi Nigel,

 

Apologies, yes I meant to mention timezones in the original email as something to consider

 

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

“addition of a timestamp” was meant to be “timezone”, my mistake.

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

From: Phelan, Nigel [mailto:nigel.phelan@...]
Sent: 28 September 2016 17:36
To: Stuart Beattie <sbeattie@...>; openmama-dev@...
Subject: RE: New feature: Extending timestamp format

 

If contemplating a new type, is it worth considering the question of time zones?  The current model doesn’t really address this, although the java constructors have some implicit assumptions that DateTime objects are stored in UTC

 

Nigel

 


Nigel Phelan | Corporate & Investment Bank | Market Data Services | J.P. Morgan

 

From: openmama-dev-bounces@... [mailto:openmama-dev-bounces@...] On Behalf Of Stuart Beattie
Sent: Wednesday, September 28, 2016 5:23 PM
To: openmama-dev@...
Subject: [Openmama-dev] New feature: Extending timestamp format

 

Hi everyone,

 

We are currently investigating an approach to improve and extend the handling of date/times in OpenMAMA.

 

The current mamaDateTime type, essentially being a 64-bit bitmask, is inflexible and has some built in limitations – such as the 4-byte second field limiting the maximum stored date to the year 2106, as well as being limited to time since the epoch.  The current datetime format is this:

 

Data Content

Size

Comments

Seconds

32 bits

Needs to be bigger to support times beyond 2106 and also has no ability to represent times before 1970.

Microseconds

20 bits

Maximum value is 1,048,575 - if nanosecond support is ever required, this would be too small (you'd need another 10 bits)

Precision

4 bits

Used to store decimal precision.

Hints

4 bits

Used to store information on whether or not to display as date / time (one of these bits going spare too, could expand to 2 with changes)

Spare (idle)

4 bits

Could be 6 if you count the values spare in hints

 

So while there are currently some bytes available to extend this somewhat, if these are used there will be no more room for expansion or new features in the existing format so we are thinking of a more long-term solution.

 

Instead we are proposing to introduce a new datetime format using a new type which is an actual struct with separate fields for the different elements (seconds, date, time etc.)  While this may be larger (in terms of storage size – eg some bits may go unused), it would have the advantages of:

a)      Being more extensible – fields could be added or changed as necessary without being bound by the current 64 bits

b)      Being more intuitive to work with

 

Format could be something like:

 

Data Content

Data Type

Comments

Seconds

int64 (yes - signed)

Not every bit will be used, but easy to work with and will support times prior to 1970.

Nanoseconds?

long int

(Current format has only microseconds)

Precision

uint8

Again, some bits going spare, but easy to work with.

Hints

uint8

Again, some bits going spare, but easy to work with.

 

 

This is in the early stages so we are asking for input as well as having a few specific questions for developers – with backwards compatibility being a major concern:

-          It is currently possible to serialise a mamaDateTime by using the native bitmask format directly – is anyone out there currently doing this and relying on this being possible?

-          For bridge developers, how do you currently encode timestamps?

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

Thanks

Stuart

 

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

 


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC


Dmitri Fedorov <dfedorov.solace@...>
 

Sorry for the late reply, for Solace Frank's option 1 is acceptable.

Regards,
Dmitri Fedorov
Software Architect
Solace, Inc.
Ottawa, ON Canada

Solace accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information provided, unless that information is subsequently confirmed in writing. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Solace Systems.

On Wed, Oct 5, 2016 at 1:16 PM, Tom Doust <tom.doust@...> wrote:

In the TREP bridge we have a wrapper around the mamaDateTime type and only use the accessor functions. There is a certain amount of messing around converting between mamaDateTime and rssl date and time representations, but so long as the set of accessor functions doesn’t change it should “just work”

 

One to bear in mind with “breaking changes” like this is that you may also introduce a dependency between the bridge version and the mama runtime version. This was (is) an issue with the change to bridge loading. If I simply install a bridge built with OpenMAMA  > 2.4 into a deployment that includes  components built with OpenMama <2.4

 

But, in principle, I agree with Frank’s #1

 

Tom

 

 

From: <openmama-dev-bounces@lists.openmama.org> on behalf of Glenn McClements <gmcclements@...>
Date: Tuesday, 4 October 2016 at 13:39
To: Nigel Phelan <nigel.phelan@...>, Frank Quinn <fquinn@...>, Stuart Beattie <sbeattie@...>, "Openmama-dev@....org" <openmama-dev@....org>


Subject: Re: [Openmama-dev] New feature: Extending timestamp format

 

For Vela we have a couple of payloads:

- V5/Wirecache has its own internal format and calls mamaDateTime setters, so this is fine.

- WombatMsg actually uses the same encoding as OpenMAMA does internally, so this will need to be changed but I’m fine with this as it was somewhat breaking encapsulation. The wire format will remain the same, we’ll just need to call a setting rather than doing a direct assignment.  

 

 

Also, from another offline thread, we should take time to consider if the new implementation will be flexible/extendable. Time zone hint and nanosecond support are two feature that spring to mind that we may like to add later.

 

These are outside of scope for the current piece of work but they have been asked about before and we’d want to make sure that any implementation could be extended to support these if needed.

 

 

GLENN MCCLEMENTS

SVP Engineering, Europe

 

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

 

From: <openmama-dev-bounces@lists.openmama.org> on behalf of "Phelan, Nigel via Openmama-dev" <openmama-dev@....org>
Reply-To: "Phelan, Nigel" <nigel.phelan@...>
Date: Tuesday, 4 October 2016 at 09:38
To: Frank Quinn <fquinn@...>, Stuart Beattie <sbeattie@...>, "openmama-dev@....org" <openmama-dev@....org>
Subject: Re: [Openmama-dev] New feature: Extending timestamp format

 

I think that, at the moment, the most important payload bridges come from Vela, Solace and Tick 42.  Would anyone from Solace and Tick 42 care to comment on the behaviour of their payload bridges and whether they took this shortcut in the implementation (I’m assuming Frank can speak authoritatively for Vela)?  Also good to know the QPID open source reference implementation gets this right.  Out of curiosity, Frank, does your ZeroMQ bridge handle this, or does it just re-use the proton payload (which would presumably automatically make it compatible)?

 

Thanks

 

Nigel

 


Nigel Phelan | Corporate & Investment Bank | Market Data Services | J.P. Morgan

 

From: Frank Quinn [mailto:fquinn@...]
Sent: Tuesday, October 04, 2016 9:13 AM
To: Stuart Beattie; Phelan, Nigel; openmama-dev@....org
Subject: RE: New feature: Extending timestamp format

 

Hi Folks,

 

The biggest concern here is around interface compatibility with the mamaDateTime type. In our types.h it’s defined as a u64* rather than a void*. Unfortunately, I expect this has given payload bridges a mechanism too tempting to ignore when it comes to simply using the underlying U64 for serialization and deserialization.

 

If payload bridge developers have treated the mamaDateTime as an encapsulated object, changing mamaDateTime to an opaque struct would be reasonably safe. Qpid proton, for example, would be unaffected by the struct change as it uses mamaDateTime_get/set.

 

However, any payload bridges which access the native U64 would need to change. Options would be either:

 

1.       Just break the bridge and move on to 6.2

2.       Make the first member of the struct the “legacy” U64 member and have a payload bridge flag to enable “backwards compatible” mode and speckle a bunch of backwards compatible code throughout the MAMA Date Time code

3.       Assume that no bridges actually use this U64 directly and risk breaking the bridge

 

Personally my vote would be for #1 - this sort of thing is the reason why we put that version handshaking code in.

 

Note I just looked through the OpenMAMA wiki for a link explaining the versioning and didn’t find one, so I just created https://github.com/OpenMAMA/OpenMAMA/wiki/OpenMAMA-Versioning for reference.

 

Cheers,

Frank

 

From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Stuart Beattie
Sent: Wednesday, September 28, 2016 5:39 PM
To: Phelan, Nigel <nigel.phelan@...>; openmama-dev@....org
Subject: Re: [Openmama-dev] New feature: Extending timestamp format

 

Hi Nigel,

 

Apologies, yes I meant to mention timezones in the original email as something to consider

 

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

“addition of a timestamp” was meant to be “timezone”, my mistake.

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

From: Phelan, Nigel [mailto:nigel.phelan@jpmorgan.com]
Sent: 28 September 2016 17:36
To: Stuart Beattie <sbeattie@...>; openmama-dev@....org
Subject: RE: New feature: Extending timestamp format

 

If contemplating a new type, is it worth considering the question of time zones?  The current model doesn’t really address this, although the java constructors have some implicit assumptions that DateTime objects are stored in UTC

 

Nigel

 


Nigel Phelan | Corporate & Investment Bank | Market Data Services | J.P. Morgan

 

From: openmama-dev-bounces@lists.openmama.org [mailto:openmama-dev-bounces@lists.openmama.org] On Behalf Of Stuart Beattie
Sent: Wednesday, September 28, 2016 5:23 PM
To: openmama-dev@....org
Subject: [Openmama-dev] New feature: Extending timestamp format

 

Hi everyone,

 

We are currently investigating an approach to improve and extend the handling of date/times in OpenMAMA.

 

The current mamaDateTime type, essentially being a 64-bit bitmask, is inflexible and has some built in limitations – such as the 4-byte second field limiting the maximum stored date to the year 2106, as well as being limited to time since the epoch.  The current datetime format is this:

 

Data Content

Size

Comments

Seconds

32 bits

Needs to be bigger to support times beyond 2106 and also has no ability to represent times before 1970.

Microseconds

20 bits

Maximum value is 1,048,575 - if nanosecond support is ever required, this would be too small (you'd need another 10 bits)

Precision

4 bits

Used to store decimal precision.

Hints

4 bits

Used to store information on whether or not to display as date / time (one of these bits going spare too, could expand to 2 with changes)

Spare (idle)

4 bits

Could be 6 if you count the values spare in hints

 

So while there are currently some bytes available to extend this somewhat, if these are used there will be no more room for expansion or new features in the existing format so we are thinking of a more long-term solution.

 

Instead we are proposing to introduce a new datetime format using a new type which is an actual struct with separate fields for the different elements (seconds, date, time etc.)  While this may be larger (in terms of storage size – eg some bits may go unused), it would have the advantages of:

a)      Being more extensible – fields could be added or changed as necessary without being bound by the current 64 bits

b)      Being more intuitive to work with

 

Format could be something like:

 

Data Content

Data Type

Comments

Seconds

int64 (yes - signed)

Not every bit will be used, but easy to work with and will support times prior to 1970.

Nanoseconds?

long int

(Current format has only microseconds)

Precision

uint8

Again, some bits going spare, but easy to work with.

Hints

uint8

Again, some bits going spare, but easy to work with.

 

 

This is in the early stages so we are asking for input as well as having a few specific questions for developers – with backwards compatibility being a major concern:

-          It is currently possible to serialise a mamaDateTime by using the native bitmask format directly – is anyone out there currently doing this and relying on this being possible?

-          For bridge developers, how do you currently encode timestamps?

-          Does anyone have any other suggestions for the new format?  For example, nanosecond precision, addition of a timestamp, or any other useful features?

 

Thanks

Stuart

 

 

STUART BEATTIE

Senior Software Engineer

 

O. +44 28909 93365

M.

sbeattie@...

 

Adelaide Exchange Building, 2nd Floor, 24-26 Adelaide Street, Belfast, BT2 8GD

velatradingtech.com | @vela_tt

 

 

 


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email


The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you. Vela Trading Technologies LLC


_______________________________________________
Openmama-dev mailing list
Openmama-dev@....org
https://lists.openmama.org/mailman/listinfo/openmama-dev