How to scale pull queues with Google Cloud Tasks

user_136598 0
Viewed 212

I have a GAE/P/Standard/FirstGen app that sends a lot of email with Sendgrid. Sendgrid sends my app a lot of notifications for when email is delivered, opened, etc.

This is how I process the Sendgrid notifications:

  • My handler processes the Sendgrid notification and adds a task to a pull queue
  • About once every minute I lease a batch of tasks from the pull queue to process them.

This works great except when I am sending more emails than usual. When I am adding tasks to the pull queue at a high rate, the pull queue refuses to lease tasks (it responds with TransientError) so the pull queue keeps filling up.

What is the best way to scale this procedure?

If I create a second pull queue and split the tasks between the two of them, will that double my capacity? Or is there something else I should consider?

====

This is how I add tasks:

q = taskqueue.Queue("pull-queue-name")
q.add(taskqueue.Task(data, method="PULL", tag=tag_name))
google-cloud-platform google-app-engine
1 Answers

I have found some information about it in Google documentation here. According to it solution for TransientError should be to:

catch these exceptions, back off from calling lease_tasks(), and then try again later.

etc.

Actually I suppose this is App Engine Task queue, not Cloud Tasks which are different product.

According to my understanding there is no option to scale this better. It seems that the some solution might be to migrate to Cloud Task and Pub/Sub which is better way to manage queues in GAE as you may find here.

I hope it will help somehow... :)

user_124103320
Related