Preconditions:
- We have a empty topic created, with a single pull subscription
- No service is actively subscribed to the subscription
- We publish about 200k messages rapidly using the
@google/pubsublibrary
Observation:
The big bump up to 2.5 million message was when we published the messages using the below code equivalent. From the log message we see that it thinks it published 200k messages.
The second small bumps were when we took the below code, but chunked the Promise.all calls with another for loop, and only gave the pubsub sdk 1000 messages at a time.
Code:
import {PubSub} from '@google-cloud/pubsub';
const pubsub = new PubSub()
const topic = pubsub.topic("some-topic");
async function publish(message) {
const dataBuffer = Buffer.from(JSON.stringify(data));
return topic.publisher.publish(dataBuffer, metadata);
}
async function processThing(thing) {
const parsed = parseThingToLotsOfThings(thing);
return (await Promise.all(
parsed.map(it => topic.publish(it))
)).length
}
async function processThings(things) {
let count = 0;
for (const thing of things) {
count += await processThing(thing);
}
console.log(`published ${count} messages`);
}
From reading the nodejs sdk source, and looking at the API reference I don't understand how this is happening.
I realize that it's a guarantee of at least once delivery, but this is an order of magnitude more, and internally the client is only including 100 messages per publish rpc call, so I don't understand why batching it in our code would change the behavior.
Is this a bug in the sdk, or should we be batching things before calling into the sdk?
