Cloud Run / Docker loading large files for ML prediction

user_4896449 0
Viewed 281

I am trying to serve a fairly large model (1GB) from a Flask powered docker container on Google Cloud Run.

I am trying to figure out the best way load the pickle / joblib model files - so as to reduce container start up time.

Since including in the github source is not an option or desired, the only idea I have now is to try and read it into the container from the Dockerfile by using gsutil.

How would this affect cold container startup time? Is there an easier way to achieve this?

Also it seems that calling joblib.load(model_file_path) in the code is not a good idea - as the load can take 30 secs or more.

google-cloud-run docker flask google-cloud-platform python
2 Answers

My recommendation is to build the container with the model inside it. You win 2 things like this:

  • At startup, you don't have to download the file (you save startup time)
  • You don't have to store the model (for information, the file are stored in /tmp directory which is a in memory file system, and thus you use memory space in your instance) and thus to increase the required memory (you save money, and the planet!)

I wrote an article on this

However, loading a 1Gb model will take time. A feature is coming (min instance, that keep a minimal number of instance warm, to prevent cold start, but it's not free!)

user_113725930

Cloud Run has an in memory file system, to download or use a 1GB file, you need to ensure you have picked enough memory for your instances. Try using 4GiB, see the docs to set memory limits

user_4816490
Related