Setting different resources for multiple Jupyter Notebook pods

user_9508556 0
Viewed 341

I recently hosted Jupyterhub in our K8s cluster. We set memory limits to 1GB so now all Jupyter notebook are with this configuration. But we have notebooks which are resource(memory and cpu) intensive and some notebooks are not resource intensive

So , is there anyway to assign different resource for multiple jupyter pods ?

I am using helm chart to deploy jupyterhub https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/installation.html

jupyterhub jupyter-notebook kubernetes
2 Answers

We use profiles to allow users to select from pods of different sizes. The pods are all deployed on large nodes, but users can select whether they get a half- or full-node pod.

As an example, we build off the daskhub configuration (full spec here):

daskhub:
  jupyterhub:
    singleuser:
      image:
      cpu:
        limit: 3.5
        guarantee: 3.5
      memory:
        limit: 22.5G
        guarantee: 22.5G
      profileList:
        - display_name: "default"
          description: "Default notebook size"
          default: true
        - display_name: "large"
          description: "Larger notebook allowance"
          kubespawner_override:
            cpu_limit: 7.0
            cpu_guarantee: 7.0
            mem_limit: 45G
            mem_guarantee: 45G

These are both spawned on 8CPU / 52GB nodes, but two of the smaller pods will go on the same node (assuming 2 users selecting "default").

user_38887190

There is no other option if your PODs are managed by Deployment or Stateful set.

Either create the Different different deployments for multiple Jupiter notebooks having different configurations or deploy each notebook as a different POD with resource requirements.

i would suggest using the VPA which will auto-scale your deployment or stateful set and adjust the resource requirement as per need.

Check more about the VPA : https://cloud.google.com/kubernetes-engine/docs/concepts/verticalpodautoscaler

user_55258240
Related