GCP pubsub: Why does publishing 200k messages rapidly result in 2.5 million messages on the topic

Viewed 659

Preconditions:

  1. We have a empty topic created, with a single pull subscription
  2. No service is actively subscribed to the subscription
  3. We publish about 200k messages rapidly using the @google/pubsub library

Observation:

Unacked message count

The big bump up to 2.5 million message was when we published the messages using the below code equivalent. From the log message we see that it thinks it published 200k messages.

The second small bumps were when we took the below code, but chunked the Promise.all calls with another for loop, and only gave the pubsub sdk 1000 messages at a time.

Code:

import {PubSub} from '@google-cloud/pubsub';

const pubsub = new PubSub()
const topic = pubsub.topic("some-topic");

async function publish(message) {
    const dataBuffer = Buffer.from(JSON.stringify(data));
    return topic.publisher.publish(dataBuffer, metadata);
}

async function processThing(thing) {
    const parsed = parseThingToLotsOfThings(thing);

    return (await Promise.all(
        parsed.map(it => topic.publish(it))
    )).length
}

async function processThings(things) {
    let count = 0;

    for (const thing of things) {
        count += await processThing(thing);
    }

    console.log(`published ${count} messages`);
}

From reading the nodejs sdk source, and looking at the API reference I don't understand how this is happening.

I realize that it's a guarantee of at least once delivery, but this is an order of magnitude more, and internally the client is only including 100 messages per publish rpc call, so I don't understand why batching it in our code would change the behavior.

Is this a bug in the sdk, or should we be batching things before calling into the sdk?

1 Answers

I suspect what is happening is that the sudden influx of 200K messages is overloading resources on the client (could be network, CPU, or thread pools). As a result, messages are getting sent to the server, but the client is too overwhelmed to process responses in a timely fashion. As a result, it ends up trying to send the messages again, resulting in the duplicates and resulting in more work for the client to do.

There are two solutions I would recommend:

  1. If possible, scale horizontally. Spread the load out over more publishers so that individual clients don't get overwhelmed.

  2. Limit the number of publishes that can be outstanding simultaneously by tracking the number of outstanding futures. The simplest way to do this is with a semaphore. Some of the Cloud Pub/Sub client libraries already support setting these limits in the library itself, e.g., Java. I imagine this is functionality that will ultimately come to the node.js library as well.

Related