Unable to limit number of Cloud Run instances spun up by Pub/Sub

Viewed 258

Situation: I'm trying to have a single message in Pub/Sub processed by exactly 1 instance of Cloud Run. Additional messages will be processed by another instance of Cloud Run. Each message triggers a heavy computation that runs for around 100s in the Cloud Run instance.

Currently, Cloud Run is configured with max concurrency requests = 1, and min/max instances of 0/5. Subscription is set to allow for 600s Ack deadline.

Issue: Each message seems to be triggering multiple instances of Cloud Run to be spun up. I believe that it is due to high CPU utilization that's causing Cloud Run to spin up additional instances to help process. Unfortunately, these new instances are attempting to process the same exact message, causing unintented results.

Question: Is there a way to force Cloud Run to only have 1 instance process a single message, regardless of CPU utilization and other potential factors?

Relevant Code Snippet:

import base64
import json

from fastapi import FastAPI, Request, Response

app = FastAPI()

@app.post("/")
async def handleMessage(request: Request):

  envelope = await request.json()

  # Basic data validation
  if not envelope:
      msg = "no Pub/Sub message received"
      print(f"error: {msg}")
      return Response(content=msg, status_code=400)

  if not isinstance(envelope, dict) or "message" not in envelope:
      msg = "invalid Pub/Sub message format"
      print(f"error: {msg}")
      return Response(content=msg, status_code=400)

  message = envelope["message"]

  if isinstance(message, dict) and "data" in message:
      data = json.loads(base64.b64decode(message["data"]).decode("utf-8").strip())

  try:
      # Do computationally heavy operations here
      # Will run for about 100s

      return Response(status_code=204)

  except Exception as e:
      print(e)

Thanks!

1 Answers

I've found the issue.

Apparently, Pub/Sub guarantees "at least once" delivery, which means it is possible for it to deliver a message to a subscriber more than once. The onus is therefore on the subscriber, which in my case is Cloud Run, to handle such scenarios (idempotency) gracefully.

Related