Slow numpy and pandas imports on Google Cloud Run

user_14085523 0
Viewed 567

I'm developping an API and deploying it on Google Cloud Run.

There is a prestart python script that import pandas and numpy. When I time the imports numpy take about 2 seconds and pandas about 4 seconds on Cloud Run as opposed to less than 0.5 second on my local machine.

I'm using python:3.8-alpine as my base image in order to build my docker container. (Though I have tried several non Alpine images... )

Here is the Dockerfile

FROM python:3.8-alpine

COPY requirements.txt ./

RUN apk add --no-cache --virtual build-deps g++ gcc gfortran make libffi-dev openssl-dev file build-base \
    && apk add --no-cache libstdc++ openblas-dev lapack-dev \ 
    && pip install --no-cache-dir uvicorn gunicorn fastapi \
    && CFLAGS="-g0 -Wl,--strip-all -I/usr/include:/usr/local/include -L/usr/lib:/usr/local/lib" \
    && pip install --no-cache-dir --compile --global-option=build_ext --global-option="-j 16" -r requirements.txt \
    && rm -r /root/.cache \
    && find /usr/local/lib/python3.*/ -name 'tests' -exec rm -r '{}' + \
    && find /usr/local/lib/python3.*/site-packages/ \( -type d -a -name test -o -name tests \) -o \( -type f -a -name '*.pyc' -o -name '*.pyo' \) -exec rm -r '{}' + \
    && find /usr/local/lib/python3.*/site-packages/ -name '*.so' -print -exec /bin/sh -c 'file "{}" | grep -q "not stripped" && strip -s "{}"' \; \
    && find /usr/lib/ -name '*.so' -print -exec /bin/sh -c 'file "{}" | grep -q "not stripped" && strip -s "{}"' \; \
    && find /usr/local/lib/ -name '*.so' -print -exec /bin/sh -c 'file "{}" | grep -q "not stripped" && strip -s "{}"' \; \
    && rm -rf /usr/local/lib/python*/ensurepip \
    && rm -rf /usr/local/lib/python*/idlelib \
    && rm -rf /usr/local/lib/python*/distutils/command \
    && rm -rf /usr/local/lib/python*/lib2to2 \
    && rm -rf /usr/local/lib/python*/__pycache__/* \
    && rm -r /requirements.txt /databases.zip \
    && rm -rf /tmp/* \
    && rm -rf /var/cache/apk/* \
    && apk del build-deps g++ gcc make libffi-dev openssl-dev file build-base 

CMD ["python","script.py"]

requirements.txt :

numpy==1.2.0
pandas==1.2.1

and the execution python file script.py :

import time

ts = time.time()
import pandas
te = time.time()
print(te-ts)

Are these slow imports to be expected? Or perhaps there is some python import trick ?

I have been looking all over stackoverflow and github issues but nothing similar to this "issue"/"behavior".

Thanks in advance.

alpine-linux pandas docker google-cloud-platform python-3.x
1 Answers

This is a known issue in the Python ecosystem.

all modules are imported at runtime, and some modules are 300-500MB large in size

There are tons of complaints about slow import times. The best thread is this one: improving speed of Python module import

As regarding Cloud Run, I have experimented with various ways, and nothing was able to reduce the slowness drastically.

If you want to use in serverless environment, or in other cold start ecosystem,
be aware the cold start can be in order of 10 seconds magnitude because of the "imports".

 importing pandas took 1.42 seconds
 importing numpy took 1.90 seconds
 importing torch took 2.84 seconds 
 importing torchvision took 0.78 seconds
 importing IPython took 1.22 seconds
 importing sklearn took 1.51 seconds
 importing import dask took 0.74 seconds

Try 1:

No solution with bumping up the CPU to maximum possible

Try 2:

No speed improvement with rewrite imports as:

pd = imp.load_module("pandas",None,"/usr/local/lib/python3.10/site-packages/pandas",('','',5))

this way the interpreter skips the "finding" phase, but still timing of this is the same, so there is no speed improvement over this.

Try 3:

No benefits of using install requirements with compile

RUN python -m pip install --no-cache-dir --compile -r requirements-prod.txt
RUN python -m compileall .

I even explored the container and the __pycache__ was built for all the modules and app code as well, but no improvement over the cold start time.

Summary:

A good read about a lazy load proposal is here

user_2437820
Related