Python ARM in Docker - Installing requirements takes ages

user_1382783 0
Viewed 596

I want to run a Docker container with some Python bots inside. These scripts need libraries like pandas, numpy and matplotlib. Everything works great in an x86 environment, but building the image on a Arm environment takes ages. From what I understood, it seems like it needs to compile the libraries for Arm instead of just downloading the compiled files. I am trying to find a way to avoid all of this.

  1. Matplotlib, pandas, numpy: They're pretty heavy packages, but I need just a few functionalities. Is the a slim version of these libraries?
  2. Any way to store the compiled stuff in a permanent cache somewhere in the pipeline? (I am using both GitHub and GitLab to build this)

Any help is appreciated Regards

github-actions gitlab-ci docker arm python
2 Answers

As @Itamar Turner-Trauring mentioned, matplotlib, numpy, pandas and many more libraries store wheels for aarch64 as well as for x86. In my case, pip was downloading the x86 wheels by default, and then it must compile them during the build and it takes ages.

After changing the pandas version I was using from 1.1.5 to 1.3.5, pip logs changed

from:

Downloading https://****/pypi/download/pandas/1.1.5/pandas-1.1.5.tar.gz (5.2 MB)

To:

Downloading https://****/pypi/download/pandas/1.3.5/pandas-1.3.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (10.9 MB)

And the entire build time changed from 58 to 8 minutes.

user_96236300

Matplotlib at least has ARM wheels (aarch64). The way I checked as going to https://pypi.org/project/matplotlib/, clicking on Download Files on the left, and looking for aarch64 wheels.

So you should be able to get wheels if you're using a sufficiently new version of pip. Make sure you do pip install --upgrade pip before installing other packages.

Second, if you do need to compile from scratch, you can cache the resulting files using BuildKit's new caching features—see here https://pythonspeed.com/articles/docker-cache-pip-downloads/. You may need to specify a different directory to cache, wherever the temporary wheels end up, but it should be possible.

user_62140340
Related