I'm trying to run XGBoost with GPU support in a docker image hosted on an AWS GPU machine (p3 family), but I keep getting:
xgboost.core.XGBoostError: [05:48:26] ../src/tree/updater_gpu_hist.cu:786: Exception in gpu_hist: [05:48:26] ../src/tree/updater_gpu_hist.cu:795: Check failed: device_ >= 0 (-1 vs. 0) : Must have at least one device
i.e. the GPU is not found by xgboost inside my docker image. The host machine runs this AWS AMI:
amzn2-ami-ecs-gpu-hvm-2.0.20210301-x86_64-ebs
which comes with NVIDIA CUDA, cuDNN, NCCL, Intel MKL-DNN, Docker, NVIDIA-Docker support. I run the docker image with the following command:
docker run --rm --gpus all --runtime nvidia <imageid> someparameter
which runs some xgboost fit with tree_method='gpu_hist' but crashes with the error message mentioned above. The code itself runs fine on the host machine, so I must be doing something stupid in my Docker image.
My docker base image is FROM ubuntu:18.04
Also when I log in to my docker image I can run nvidia-smi and the GPU seems to be detected correctly, but XGBoost doesn't seem to find it.
I thought that with the particular AMI I use '--runtime nvidia' when running docker should take care of loading all required libs for GPU support, but seems not to be the case. I'm a docker beginner so I might be missing something obvious. Also I couldn't find a public docker image with xgboost w/ GPU support, but if someone can point me to one, that would be great too.