We're running a standard Ubuntu 20 image with NVIDIA drivers and nvidia docker installed.
Everything works just fine.
Eventually the dockers using GPU need to be updated, then all of sudden, NVIDIA drivers disappear, docker cannot start nvidia runtime and nvidia-smi complains about driver not being loaded.
...then we have to download and reinstall nvidia drivers again.
I wonder if internally GCP VMs deallocate GPU resources when they're not being used, since during the short period GPU is not being used (during update) nothing is loaded at GPU.