Apache Pulsar send message seems slow

user_2017457 0
Viewed 254

I'm making some tests with Apache Kafka, RabbitMQ and Apache Pulsar, using C# client libraries. Tonight I started to do some tests with producer of messages for Apache Pulsar, and using the official .net library (DotPulsar) I see that the sending of messages is very slow compared to the other ones. It seems impossibile that Pulsar is so much slower than Kafka and RabbitMQ, I don't know if I am doing something wrong. I just wrote:

    const string myTopic = "persistent://public/default/mytopic";

    await using var client = PulsarClient.Builder()
      .ServiceUrl(new Uri("pulsar://172.16.50.174:6650"))
      .Build();

    await using var producer = client.NewProducer()
                                     .Topic(myTopic)
                                     .Create();

    for (int i = 0; i < 1000; i++) {
      var ret = await producer.Send(UTF8Encoding.UTF8.GetBytes($"Hello World {i:00000}"));
    }

It needs minutes go sent 10000 messages, I don't understand why. Consuming instead is very fast as expected. Can you help me on this?

Thank you

apache-pulsar .net c#
2 Answers

The problem here is awaiting on each iteration. Basically the send returns task that will await 2 operations: sending and waiting for confirmation from broker. And while sending is usually very fast, waiting for acknowledgment takes quite a lot of time.

As the author of Pulsar.Client library I've described some ideas here. To sum up you don't need to use await in a loop, rather call send and await the result in another thread (by using ContinueWith or Queue for tasks) and also enable batching. There is one more case that Pulsar.Client covers - if you don't care about the broker acknowledgement and just want send as fast as you can but still respecting producer internal queue size of pending messages - you can use SendAndForgetAsync method this will allow sending messages with await in the loop

user_17806480

Based on your comments I tried to do something like

await Task.WhenAll(messages.Select(m => producer.SendAsync(m)));

and I see the speed is now fast as I expected. Is it correct way? Awaiting all the tasks in this way am I getting the acks of all the messages?

user_20174570
Related