GCP - GPU staging time reduction

Admin 1
Viewed 113

I have an application that requires the smallest boot-time/TTL possible with GPUs attached to a VM in GCP CE. To keep cost down, my infrastructure is dependent on starting and stopping instances as demand increases/decreases.

I have achieved sub-5second start times with custom images without GPUs, but as soon as I attach a GPU, the time to "RUNNING" is always past 20-30s.

I have tried multiple different distros, clear linux, prepackaged Nvidia driver images, minimal installs of Fedora, minimalised Debian, reductions to kernel and userspace - systemd-analyze says my boot-time is 3s, but starting the VM with a GPU takes 20-30s in "STAGING" before running.

This only occurs when the gpu is attached to the VM and when removed the VM starts within the time mentioned by systemd-analyze. It is consistent across all distros and bootimages.

Is there any packages or documentation I am missing to speed up this staging-time with a GPU attached or is this a limitation with GCP's internal staging of GPU instances?

I'd much appreciate any help or advice.

If you're also experiencing this issue and would like to track its progress, I created a issue report: https://issuetracker.google.com/issues/200575905

boot google-compute-engine google-cloud-platform gpu
1 Answers

This is an internal limitation in GCE and GKE, there's not a lot that can be currently done to remediate this.
However, I noticed that startup times have dropped over time, so there is some improvement in this matter. You can report this via Public Issue Tracker to follow development.

You can also consider using Committed Use Discount or Sustained Use Discounts. It may be beneficial in the long run to keep the instances running and therefore avoid the startup problem altogether.

user_146634970
Related