Why does starting daskdev/dask into a Pod fail?

user_562440 0
Viewed 288

Why does kubectl run dask --image daskdev/dask fail?

# starting the container with docker to make sure it basically works
➜  ~ docker run --rm -it --entrypoint bash daskdev/dask:latest
(base) root@5b34ce038eb3:/# python
Python 3.8.0 (default, Nov  6 2019, 21:49:08) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dask
>>> 
>>> exit()
(base) root@5b34ce038eb3:/# exit
exit

# now trying to fire up the container on a minikube cluster
➜  ~ kubectl run dask --image daskdev/dask  
pod/dask created

# let's see what's going on with the Pod
➜  ~ kubectl get pods -w
NAME                              READY   STATUS             RESTARTS   AGE
dask                              0/1     CrashLoopBackOff   1          13s
dask                              0/1     Completed          2          24s
dask                              0/1     CrashLoopBackOff   2          38s

# not sure why the logs look like something is missing
➜  ~ kubectl logs dask --tail=100
+ '[' '' ']'
+ '[' -e /opt/app/environment.yml ']'
+ echo 'no environment.yml'
+ '[' '' ']'
+ '[' '' ']'
+ exec
no environment.yml
dask-kubernetes dask kubernetes
1 Answers
  1. So basically, if you will check result of kubectl describe pod dask, you will see that last state was Terminated with Exit Code 0, that literally means you container was launched successfully, did it job and finished also successfully. What else you expect to happen with pod? IN addition, when you create pod using kubectl run dask --image daskdev/dask- it creates with the restartPolicy: Always by default!!!!

Always means that the container will be restarted even if it exited with a zero exit code (i.e. successfully).

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 02 Apr 2021 15:06:00 +0000
      Finished:     Fri, 02 Apr 2021 15:06:00 +0000
    Ready:          False
    Restart Count:  3
    Environment:    <none>
  1. There is no /opt/app/environment.yml in your container. If im not mistake, you should first configure it with prepare.sh. PLease check more here - DASK section
#docker run --rm -it --entrypoint bash daskdev/dask:latest
(base) root@431d69bb9a80:/# ls -la /opt/app/
total 12
drwxr-xr-x 2 root root 4096 Mar 27 15:43 .
drwxr-xr-x 1 root root 4096 Mar 27 15:43 ..

not sure why the logs look like something is missing ➜ ~ kubectl logs dask --tail=100 ... exec no environment.yml

  1. There is already prepared helm DASK chart. Use it. It works fine:
helm repo add dask https://helm.dask.org/
helm repo update
helm install raffael-dask-release dask/dask

NAME: raffael-dask-release
LAST DEPLOYED: Fri Apr  2 15:43:38 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing DASK, released at name: raffael-dask-release.

This release includes a Dask scheduler, 3 Dask workers, and 1 Jupyter servers.

The Jupyter notebook server and Dask scheduler expose external services to
which you can connect to manage notebooks, or connect directly to the Dask
cluster. You can get these addresses by running the following:

  export DASK_SCHEDULER="127.0.0.1"
  export DASK_SCHEDULER_UI_IP="127.0.0.1"
  export DASK_SCHEDULER_PORT=8080
  export DASK_SCHEDULER_UI_PORT=8081
  kubectl port-forward --namespace default svc/raffael-dask-release-scheduler $DASK_SCHEDULER_PORT:8786 &
  kubectl port-forward --namespace default svc/raffael-dask-release-scheduler $DASK_SCHEDULER_UI_PORT:80 &

  export JUPYTER_NOTEBOOK_IP="127.0.0.1"
  export JUPYTER_NOTEBOOK_PORT=8082
  kubectl port-forward --namespace default svc/raffael-dask-release-jupyter $JUPYTER_NOTEBOOK_PORT:80 &

  echo tcp://$DASK_SCHEDULER:$DASK_SCHEDULER_PORT               -- Dask Client connection
  echo http://$DASK_SCHEDULER_UI_IP:$DASK_SCHEDULER_UI_PORT     -- Dask dashboard
  echo http://$JUPYTER_NOTEBOOK_IP:$JUPYTER_NOTEBOOK_PORT       -- Jupyter notebook

NOTE: It may take a few minutes for the LoadBalancer IP to be available. Until then, the commands above will not work for the LoadBalancer service type.
You can watch the status by running 'kubectl get svc --namespace default -w raffael-dask-release-scheduler'

NOTE: It may take a few minutes for the URLs above to be available if any EXTRA_PIP_PACKAGES or EXTRA_CONDA_PACKAGES were specified,
because they are installed before their respective services start.

NOTE: The default password to login to the notebook server is `dask`. To change this password, refer to the Jupyter password section in values.yaml, or in the README.md.
  1. If you still want create manually pod, use below... Main idea is set restartPolicy: Never.
apiVersion: v1
kind: Pod
metadata:
  name: dask-tesssssst
  labels:
    foo: bar
spec:
  restartPolicy: Never
  containers:
  - image: daskdev/dask:latest
    imagePullPolicy: Always
    name: dask-tesssssst
  1. Please check DASK KubeCluster official documentation for more examples. Last one I took exactly from there.
user_99290150
Related