Docker Swarm does not support use of GPU devices

user_16270072 0
Viewed 1321

I am trying to deploy GPU workload into my swarm cluster.

My service manifest looks like this.

nvidia:
    image: "tensorflow/tensorflow:devel-gpu"
    volumes:
      - type: bind
        source: ./ubuntu_gpu
        target: /app
        consistency: cached
    build:
      context: ./ubuntu_gpu
    ports:
      - 6186:80
    working_dir: "/app"
    runtime: nvidia
    environment:
      - LD_LIBRARY_PATH=/usr/local/cuda-11.3/targets/x86_64-linux/lib
   deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            capabilities: [gpu]

The approach described above was taken from docker official documentation on GPU support https://docs.docker.com/compose/gpu-support/

When I deploy the stack, I get this error below.

Additional property devices is not allowed

It seems like docker swarm does not have visibility of the GPU device attached to the worker node.

Does anyone know if this feature works only docker-compose but not docker-swarm or if there is a work around this ?

nvidia-docker docker-swarm docker-compose gpu
1 Answers

At the time of writing, docker swarmkit does not seem to support devices, see this issue on the docker github, there is quite the longstanding discussion around it.

user_161748360
Related