I have a need that does not quite fit Argos or Airflow in that I would like to combine many containers into a pod and limit the number of containers started at once. The task is to kick off aggregation scripts (scrapy/crawlers, API calls;etc.). These scripts feed a separate system using a replica set to process the pulled data. I need to limit the number of containers running at once as there will likely be many running in the future. This isn't quite the use case for a DAG. These scripts have 1 task and I feel that combining 250 aggregators (or a divided amount) in a DAG with concurrency limits is a bit overkill for my task. I cannot seem to find the answer through a Google query. Is there a way to limit the number of containers executing at once? Each container runs a command to kick off the aggregator and terminates once the script finishes.
1 Answers
Answer
No, there is no way to tell Kubernetes to manage the number of containers running in a Pod. This is purely up to the user when they define their Deployments/Daemonsets/Jobs etc.
Taken from the docs:
Pods are the smallest deployable units of computing that you can create and manage in Kubernetes.
You can, however, manage batch workloads through Kubernetes Job resources which I will link here.
Jobs can specify the amount of parallelism at Pod granularity.
All this is to say that you would have to implement your workload as containers into separated Pods as is required by your use case. These Pods could talk to each other through ClusterIP Services. You can also keep tightly-coupled containers in the same Pod where they can talk over localhost.
Related