Training using object detection api is not running on GPUs in AI Platform

user_4274552 0
Viewed 304

I am trying to run the training of some models in tensorflow 2 object detection api.

I am using this command:

gcloud ai-platform jobs submit training segmentation_maskrcnn_`date +%m_%d_%Y_%H_%M_%S` \
    --runtime-version 2.1 \
    --python-version 3.7 \
    --job-dir=gs://${MODEL_DIR} \
    --package-path ./object_detection \
    --module-name object_detection.model_main_tf2 \
    --region us-central1 \
    --scale-tier CUSTOM \
    --master-machine-type n1-highcpu-32 \
    --master-accelerator count=4,type=nvidia-tesla-p100 \
    -- \
    --model_dir=gs://${MODEL_DIR} \
    --pipeline_config_path=gs://${PIPELINE_CONFIG_PATH}

The training job is submitted successfully but when I look at my submitted job on AI platform I notice that it's not using the GPUs! enter image description here

Also, when looking at the logs for my training job, I noticed that in some cases it couldn't open cuda. It would say something like this:

Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib64

I was using AI platform for training a few months back and it was successful. I don't know what has changed now! In fact, for my own setup, nothing has changed.

For the record, I am training Mask RCNN now. A few months back I trained Faster RCNN and SSD models.

gcp-ai-platform-training google-ai-platform object-detection-api object-detection tensorflow
1 Answers

Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib64

I'm not sure as I couldn't test anyhow. With a quick google search, It appeared that people have encountered this issue for many reasons and the solution is some kind of depends. In SO, there is the same query was asked, and you probably missed it somehow, check it first, here.

Also, check this related issue posted below

After checking with every possible solution, and still remains the issue, then update your query with it.

I think there some mismatches in your Cuda version (CUDA, cuDNN) and tf version, you should check them first in your working environment. Also, ensure you update the Cuda path properly. According to the given error message, you need to make it ensure that the following set properly.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.0/lib64/
user_92157800
Related