How to manage GPU memory usage with docker?

Viewed 17

i have a problem of GPU memory usage on my app (i have a RTX2080, 8GB Vram). My app is dockerize (1 docker for front, 2 docker for 2 API of AI based on Dockerhub Tensorflow, 1 docker for Mongo DB)

When i start one of my API (search of faces in a video), nvidia-smi shows me among other things:

0   N/A  N/A     53044      C   /usr/local/bin/python            6061MiB

Well ok, my API is using all my GPU RAM. At the end, i do:

    del device
    del model
    gc.collect()
    torch.cuda.empty_cache()
    logger.info(torch.cuda.mem_get_info())

My log shows (969342976, 8366915584) -> That's seems good in my docker!

But nvidia-smi still shows that my API had a GPU memory usage of ~6000 Mb

And if i try to launch my second API, i have a cuda out of memory error. Is it possible that it is docker which keeps in memory the GPU used?

I hope i'm clear, thanks for help!

0 Answers
Related