What is the best way to handle changes to the filter on a subscription in Google Cloud Pub/Sub?

user_11783289 0
Viewed 395

Problem: I know that Google Pub/Sub subscription cannot be patched to update the filter. I am just figuring out other ways to handle updates to filter in production.

Approach I could come up with:

  1. Modify the push config to pull in existing subscription (old filter) so that it accumulates messages
  2. Create a new subscription with latest filter
  3. Transfer the messages from old subscription to a topic using dataflow
  4. Detach the old subscription from the topic

Problems I see with the approach:

  1. As both subscriptions exist at a point of time, I could end up processing duplicate messages

Any suggestions on the best way to handle this?

google-cloud-pubsub google-cloud-platform
2 Answers

The timing is important to minimize the duplicates or the message loss.

Firstly, I will deploy a service (Cloud Run for exemple) that save the PubSub message as is somewhere (in a Cloud Storage file, in bigQuery, in Firestore,...)

Then, and in the same time, I will change the push of the old subscription to push to my Cloud Run service; and create the new push subscription with the new filter

Finally detach the subscription.

If you have the capacity, in your REAL app to detect the message already processed, you can remove them from your save location (it's easier with BigQuery for example) and then reprocess only the missing messages. With dataflow, or manually

However, it's recommended to have idempotent processing of your message. Keep in mind that PubSub is at least 1 delivery message and even with the same subscription you could have duplicates.

user_113725930

As you note, a filter expression cannot be changed once a subscription has been created. To effectively change a filter, please do the following steps:

  1. Create a snapshot on the existing subscription (subscription A).
  2. Create a new subscription (subscription B) with the new filter expression.
  3. Seek subscription B to the snapshot created in step 1.
  4. Change your subscribers to use subscription B instead of A.

Please note that during the time subscribers are switching from subscription A to B, there will be a high rate of duplicates delivered as messages are acked independently on the two subscriptions.

If you need to minimize the duplicates processed and can afford downtime, you can stop the all subscribing jobs/servers before step 1 and restart them configured to pull from the new subscription, B, after step 4. All steps must be completed well within the configured message_retention_duration to prevent message loss.

user_138792210
Related