Re: C# DQPublisher Manager


Mathias Kim
 

Hi Frank,

 

I tested the broker on a Fedora Virtualbox and the results were much better than on my Windows version. The performance was as good as with point to point for capturereplayc/mamalistenc (5 Symbols) and our C# classes.

For testing the C# classes I was simulating orderbooks. In the end I was sending an average of 14 messages per second. Each message contained on average 25 fields.

The same test with a Windows broker led to huge delay. Since the amount of data is fairly low, something must have gone wrong when compiling qpid under Windows.

Interestingly, also under Fedora I have to add the –t parameter. Without, no communication happens.

 

Unfortunately, with Virtualbox/Fedora our C# publisher and listener experience some problems since not every subscription is successful. In about 50% of the cases I get this behavior:

-          Publisher connects successfully to qpid broker

-          Listener connects successfully to qpid broker and submits desired symbol

-          Symbol arrives at publisher, publisher starts sending data to qpid broker

-          Qpid broker receives data (as I can see on the output) but never forwards to the listener

Monitoring the port shows that no messages ever arrive at the machine the listener is running on.

No similar behavior with capturereplayc/mamalistenc. Also, no similar issues with qpid under Windows.

 

Did you test the qpid broker/openmama combo with a C# Listener yet?

 

Regards

 

Mathias

 

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Montag, 22. Juni 2015 17:40
To: Mathias Kim
Cc: openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

 

I was only testing with producer / consumer which is only one symbol so perhaps that's part of the issue here. I'll test with more symbols too when I get back to my machine. What sort of data rates were you running at?

 

Cheers,

Frank

 

On Mon, Jun 22, 2015 at 4:25 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

thanks for the answer.

 

I tried using Proton 0.8 but that changed nothing.

Also, I checked the queue depth. Publisher and Subscriber app had a depth of 0.

In addition, I checked the broker with capturereplayc and mamalistenc. More or less the same results: with one symbol there was no visible latency. After subscribing to 5 symbols I saw the same behavior as with our C# Publisher/Subscriber. So I think the issue lies with the Windows Broker. Also, I was running it on a Server within my network and the delay got even larger.

So I think the delay is caused by the (Windows) Qpid broker.

 

Since I have no clue how to modify the Windows Qpid broker, I’m going to test it with a Fedora Virtual Machine.

 

Regards,

 

Mathias

 

 

 

From: Frank Quinn [mailto:fquinn.ni@...]
Sent: Samstag, 20. Juni 2015 13:09
To: Mathias Kim
Cc:
openmama-dev@...
Subject: Re: [Openmama-dev] C# DQPublisher Manager

 

Hi Mathias,

I just tried an equivalent test with Qpid Broker 0.32 and proton 0.9.1 (with a few small changes to let it compile as proton changed their interface again) and the producer keeps crashing... so we'll get back to what's going on there later.

So I tried with Proton 0.7 on linux (Fedora 22) and I can see:

1. Logging doesn't need to be disabled - seems to work regardless.

2. Performance is pretty much the same with the broker as it is point to point

3. CPU is spread across several cores

4. Latency is around 4ms at data rates of around 10,000 msg/s

If you're getting latencies in the 2 second region, my first guess is that you're queuing in your client, have you tried printing out the queue size in your application? This is all finger in the air stuff though because this is all running on my local laptop.

When you're operating point to point, the producer is essentially throttled by the speed of the consumer. When a broker is involved, this relationship is effectively decoupled, so you may find that the producer is actually publishing faster than you expect now (enqueuing for sending is not the same as actually sending). That or else your applications haven't been compiled with multithreading support and they're all just fighting with each other for scheduler time. I'll try a windows test when I get a chance to see if I can replicate what you're seeing.

What sort of data rates and message sizes were you putting through?

 

Cheers,

Frank

 

On Tue, Jun 16, 2015 at 4:31 PM, Mathias Kim <Mathias.Kim@...> wrote:

Hi Frank,

 

Testing the broker feature took us a while since building Qpid isn’t as straightforward on Windows as it seems on Linux.

As you suggested when using the broker the slowdown after client disconnect disappears. However, the broker seems to be very slow.

 

We were testing it with the same C# classes that showed acceptable performance without the broker. Using the broker, every received message adds a delay of roughly two seconds at the client app. The publisher app is not affected and works as fast as before. This only occurs if we simulate a very busy symbol. Symbols with fewer updates are not affected.

Also, Qpid only uses one core whereas the publisher/client setup without broker uses all 4 available cores evenly.

 

How is the performance on Linux?

 

We’re using

-          Windows 8

-          Broker, publisher and client on the same machine

-          Qpid 0.32 & Proton 0.7

 

Also, we noticed a very strange thing: this whole thing only works if we enable all logging(-t). Without, no communication happens.

 

Regards

 

Mathias

 

 

Join Openmama-dev@lists.openmama.org to automatically receive all group messages.