I'm working on a microservice feature from scratch and have to make some design choices.
My needs are :
- be able to periodically run a batch job (CronJob) that aggregates data
- then send the data set to a service for processing
- have resiliency over service failure (pod rescheduled, pod crashing, ..)
- the data processing involve long database queries and even Big Queries, so parallelization is required
I came up with an architecture in two parts, composed of a Kubernetes CronJob doing the initial data aggregation, and publishing messages into a queue. Then this queue is consumed by the service itself, inside a Go routine, so that the load is shared across all instances (pods) of the service, and the resiliency is ensured.
Now I am stuck in a decision dilemma between :
- decoupling the queue consumer from the service in separate pods
- or keep the queue consumer in the same component (in the sense the executable) as the service
While on one side I find it more logic in terms of architecture to decouple the consumer from the service (separate executables, separate pods), for example, so both can scale independently inside the cluster, on the other side the initial wish of introducing a queue in the design was to have resiliency over service failures and the ability to dispatch the load across instances that will consume messages, and that looks overkill to me.
I didn't choose the simple option of a CronJob doing API calls to the service because only the service instance receiving the call will do the data processing.
Is it a non-sense to have a service component that also runs a queue consumer, knowing that the service will have multiple instances of itself, in terms of design?
What kind of advantage or disadvantage do I have if I decouple both or keep them coupled?