Why does it take 5 seconds to import numpy in fresh virtualenv?

user_15045544 0
Viewed 258

Background

Hello. We write python code that gets run on servers we don't control. We don't know very much about the environment the code gets run in. Our code gets rejected if it takes more than 3 seconds to run. So I decided to start timing our code using a virtual environment, to give a worst-case scenario runtime estimate.

Question

I was surprised to find that in a fresh virtual environment, it takes 5 seconds just to import numpy. Can someone explain why that would be? Is there some trick to speed it up? It seems prohibitive for a basic library to be that slow.

Steps to reproduce:

  1. rm -r /tmp/env
  2. virtualenv /tmp/env
  3. /tmp/env/bin/pip install numpy
  4. time /tmp/env/bin/python3 -c 'import numpy'

On my high-end laptop it returns

  real  0m5.534s
  user  0m0.612s
  sys   0m0.133s

Note that, for reasons that are not clear to me, if you repeat the step #4 without going back to step #1, the time is significantly less. This persists even if you run, say, find /tmp -type d -name '__pycache__' -exec rm -r {} \; first. It must be some other type of module caching that I don't know about? Anyway, it doesn't matter for my purposes, because I can't control the cache state of the remote servers.

virtualenv numpy python
1 Answers

Because numpy is soo huge, the first ever import would take very long, not only loading all those files but "compiling" them all into .pyc, once that that is done future import are more faster. That is why when you start anew it take soo long versus when you only do your step #4 without going back to #1

As for trick to speed it up, looking at pip install -h the --compile option should do the trick, I think...

user_56449610
Related