Specifics of using a push subscription as a load balancer

user_3043930 0
Viewed 271

I am trying to send IoT commands using a push subscription. I have 2 reasons for this. Firstly, my devices are often on unstable connections so going through the pubsub let me have retries and I don't have to wait the QoS 1 timeout (I still need it because I log it for later use) at the time I send the message. The second reason is the push subscription can act as a load balancer. To my understanding, if multiple consumers listen to the same push subscription, each will receive a subset of the messages, effectively balancing my workload. Now my question is, this balancing is a behavior I observed on pull subscriptions, I want to know if:

  1. Do push subscription act the same ?
  2. Is it a reliable way to balance a workload ?
  3. Am I garanteed that these commands will be executed at most once if there is, lets say, 15 instances listening to that subscription ?

Here's a diagram of what I'm trying to acheive: enter image description here

Idea here is that I only interact with IoT Core when instances receive a subset of the devices to handle (when the push subscription triggers). Also to note that I don't need this perfect 1 instance for 1 device balancing. I just need the workload to be splitted in a semi equal manner.

EDIT: The question wasn't clear so I rewrote it.

google-cloud-pubsub google-cloud-platform
1 Answers

I think you are a bit confused about the concepts behind Pub/Sub. In general, you publish messages to a topic for one or multiple subscribers. I prefer to compare Pub/Sub with a magazine that is being published by a big publishing company. People who like the magazine can get a copy of that magazine by means of a subscription. Then when a new edition of that magazine arrives, a copy is being sent to the magazine subscribers, having exactly the same content among all subscribers.

For Pub/Sub you can create multiple push subscriptions for a topic, up to the maximum of 10,000 subscriptions per topic (also per project). You can read more about those quotas in the documentation. Those push subscriptions can contain different endpoints, in your case, representing your IoT devices. Referring back to the publishing company example, those push endpoints can be seen as the addresses of the subscribers.

Here is an example IoT Core architecture, which focuses on the processing of data from your devices to a store. The other way around could also work. Sending a message (including device/registry ID) from your front-end to a Cloud Function wrapped in API gateway. This Cloud Function then publishes the message to a topic, which sends the message to a cloud Function that posts the message using the MQTT protocol. I worked out both flows for you that are loosely coupled so that if anything goes wrong with your device or processing, the data is not lost.

Device to storage:

  1. Device
  2. IoT Core
  3. Pub/Sub
  4. Cloud Function / Dataflow
  5. Storage (BigQuery etc.)

Front-end to device:

  1. Front-end (click a button)
  2. API Gateway / Cloud Endpoints
  3. Cloud Function (send command to pub/sub)
  4. Pub/Sub
  5. Cloud Function (send command to device with MQTT)
  6. Device (execute the command)
user_79505920
Related