Google PubSub throttling

user_4845952 0
Viewed 171

I have a chain of data pipelines communicating to each other through pipeline-specific Pub/Sub topics.

[pipeline #1] -> [pipeline #2] -> [pipeline #3] -> ...

Each pipeline is totally stateless and can scale horizontally based on the respective pub/sub queue size.

Each pipeline has its own throughput.
The pipeline #2 uses a 3rd party API. This API has hourly/daily rate limit.

Sometimes I have spikes in the workload. All pipelines but API pipelines can scale up to handle the workload. But not the API pipeline. It can eventually catch up, but not right away.

Is there a way to throttle down a pub/sub topic/subscription throughput to a specific number of messages per hour/day?

Pretty sure the answer is NO. If so, what are my other options?

google-cloud-pubsub google-cloud-platform
1 Answers

You are right, the answer is no. Cloud Task has that throttling feature (named rate limit).

One solution design could be:

  1. PubSub invokes the next step
  2. Processing (Cloud Functions/Cloud Run/Something else) that stack the request in Cloud Task
  3. Cloud Task invokes the next step
  4. Processing (Cloud Functions/Cloud Run/Something else) that call your 3rd party API

Quite dirty, but you haven't other built-in solution on Google Cloud.

user_113725930
Related