How to limit the amount of simultaneously running jobs of a certain "type"?

user_18775 0
Viewed 312

I would like to be able to limit the amount of jobs of a given "type" that run at the same time (maybe based on their label, e.g. no more than N jobs with label mylabel may run at the same time).

I have a long running computation that requires a license key to run. I have N license keys and I would like to limit the amount of simultaneously running jobs to N. Here's how I imagine it working: I label the jobs with some special tag. Then, I schedule N + K jobs, then at most N jobs may be in state "running" and K jobs should be in the queue and may only transition to "running" state when the total number of running jobs labeled mytag is less or equal to N.

[UPDATE]

  • The jobs are independent of each other.
  • The execution order is not important, although I would like them to be FIFO (time wise).
  • The jobs are scheduled on user requests. That is, there is no fixed amount of work known in advance that needs to be processed, the requests to run a job with some set of parameters (configuration file) come sporadically in time.
kubernetes
2 Answers

Unfortunately there is no built-in feature to do it using labels in k8s. But since your jobs are scheduled based on unpredictable user requests, you can achieve your goal like this:

  • create a new namespace kubectl create namespace quota-pod-ns
  • create a ResourceQuota
apiVersion: v1
kind: ResourceQuota
metadata:
  name: pod-max-number
  namespace: quota-pod-ns
spec:
  hard:
    pods: "5"

This will limit max number of pods to 5 in the namespace quota-pod-ns.

  • create k8s jobs in the quota-pod-ns namespace.

When you want to run the 6th job in that namespace, k8s will try to create the 6th pod and will fail to do that. But once one of the running pods is Completed, job-controller will create that new pod within a max limit.

user_177394400

You can ‘parallelism’ attribuate of the job, but K8s won’t spin any jobs till the current active be less than parallel threshild

user_8714100
Related