Docker restart python script on failure

Jakub Mitura 0
Viewed 38

I have the docker container that executes the python file - I want it to restart on the failure of the script - usually memory errors.

Docker is running, python file works but after script failure container just exit.

The container just --restart always policy does not work - what am I doing wrong?

docker command:

sudo docker run --init --gpus all \
    --ipc host --privileged --net host \
    -p 8888:8888 -p49053:49053 
    --restart always \
    -v /mnt/disks/sde:/home/sliceruser/data \
    -v /mnt/disks/sdb:/home/sliceruser/dataOld \
    slicerpicai:latest

end of docker file

ENTRYPOINT [ "/bin/bash", "start.sh","-l", "-c" ]

start.sh

cd /home/sliceruser/data/piCaiCode
git pull
python3.8 /home/sliceruser/data/piCaiCode/Three_chan_baseline_hyperParam.py
docker python
1 Answers

He restart policy works, you might just not see it. I suggest to check the number of retries so far on the container:

docker inspect -f "{{ .RestartCount }}" my-container

Also docker tries indefinitely (with --retry always) but it does wait always longer if the start keeps failing.

If you say your script has memory issues, it would be good to address those before looking at issues with Docker. if the reason why the container stops lies outside docker, that obviously stops the container from restarting as well. So I recommend checking the container logs and thinking of what you do in order to manually restart the container after a failure.

For more details check also the official reference of docker run

If you want to reproduce what I wrote above do the following:

Open 1 terminal and run:

docker stats

Open a second terminal and run:

docker run -d --name testcontainer --restart always alpine:latest sh -c "sleep 5 && exit 2"

This will start a container that "crashes" every 5s.

In the same terminal run:

# check the status and see how it waits longer and longer to restart
docker container ls --filter name="testcontainer"

# check the number of restarts so far
docker inspect -f "{{ .RestartCount }}" testcontainer

Friendly footnote: I think you are lucky it doesn't restart because this is such an unsecure container. ;)

useruGkL0
Related