i have a problem of GPU memory usage on my app (i have a RTX2080, 8GB Vram). My app is dockerize (1 docker for front, 2 docker for 2 API of AI based on Dockerhub Tensorflow, 1 docker for Mongo DB)
When i start one of my API (search of faces in a video), nvidia-smi shows me among other things:
0 N/A N/A 53044 C /usr/local/bin/python 6061MiB
Well ok, my API is using all my GPU RAM. At the end, i do:
del device
del model
gc.collect()
torch.cuda.empty_cache()
logger.info(torch.cuda.mem_get_info())
My log shows (969342976, 8366915584) -> That's seems good in my docker!
But nvidia-smi still shows that my API had a GPU memory usage of ~6000 Mb
And if i try to launch my second API, i have a cuda out of memory error. Is it possible that it is docker which keeps in memory the GPU used?
I hope i'm clear, thanks for help!