I'm very lost on how to solve my particular problem, which is why I followed the getting help guideline in the Object Detection API and made a post here on Stack Overflow.
To start off, my goal was to run distributed training jobs on Azure. I've previously used gcloud ai-platform jobs submit training with great ease to run distributed jobs, but it's a bit difficult on Azure.
I built the tf1 docker image for the Object Detection API from the dockerfile here.
I had a cluster (Azure Kubernetes Service/AKS Cluster) with the following nodes:
4x Standard_DS2_V2 nodes
8x Standard_NC6 nodes
In Azure, NC6 nodes are GPU nodes backed by a single K80 GPU each, while DS2_V2 are typical CPU nodes.
I used TFJob to configure my job with the following replica settings:
Master (limit: 1 GPU) 1 replica
Worker (limit: 1 GPU) 7 replicas
Parameter Server (limit: 1 CPU) 3 replicas
Here's my conundrum: The job fails as one of the workers throw the following error:
tensorflow/stream_executor/cuda/cuda_driver.cc:175] Check failed: err == cudaSuccess || err == cudaErrorInvalidValue Unexpected CUDA error: out of memory
I randomly tried reducing the number of workers, and surprisingly, the job worked. It worked only if I had 3 or less Worker replicas. Although it took a lot of time (bit more than a day), the model could finish training successfully with 1 Master and 3 Workers.
This was a bit vexing as I could only use up to 4 GPUs even though the cluster had 8 GPUs allocated. I ran another test: When my cluster had 3 GPU nodes, I could only successfully run the job with 1 Master and 1 Worker! Seems like I can't fully utilize the GPUs for some reason.
Finally, I ran into another problem. The above runs were done with a very small amount of data (about 150 Mb) since they were tests. I ran a proper job later with a lot more data (about 12 GiB). Even though the cluster had 8 GPU nodes, it could only successfully do the job when there was 1 Master and 1 Worker.
Increasing the Worker replica count to more than 1 immediately caused the same cuda error as above.
I'm not sure if this is an Object Detection API based issue, or if it is caused by Kubeflow/TFJob or even if it's something Azure specific. I've opened a similar issue on the Kubeflow page, but I'm also now seeing if I can get some guide from the Object Detection API community. If you need any further details (like the tfjob yaml, or pipeline.config for the training) or have any questions, please let me know in the comments.