First import of python modules slow on a new VM, even after warming __pycache__?

I have an Amazon Linux image that I use for some ML tasks, which use Python heavily. When I spin up a new machine, the first time I import all the python packages (numpy, scipy, scikit-learn, xarray, etc.) take a long time-- if I import absolutely everything, it's about 80 seconds. Subsequent imports, even in separate Python processes on the same VM, are extremely fast: 1-2 seconds for everything.

I'd like to eliminate that minute-plus on startup if I could; I only pay the performance cost once per VM, but the VMs are short-lived (typically for one job, perhaps 45-60 minutes).

My original assumption was that this was related to __pycache__; that the minute the first time was compiling and caching the bytecode of all the libraries, and the subsequent runs were faster because all that was already finished. So I added a script to my image builder, that imported every single module I ever reference as part of creating the image. However, the results were not all as I expected, and there was no performance gain:

1. Running the script as part of the image builder is almost instant (I was not expecting this).
2. Running the script does create the expected __pycache__ files alongside their sources in site-packages.
3. The __pycache__ files exist on systems created from the image, before any python has ever been run.
4. Running the script on a system newly created from the image still takes over a minute, even though the __pycache__ files are already in place.
5. Running the script on the same system a second time takes under a second.
6. <module>.__cached__ on the system created from the image appears to refer to the correct .pyc file in the correct __pycache__ folder that was in place from the image.
7. The dates of all the .pyc files in the __pycache__ folders reflect the time of image creation, not of VM instantiation.

So it seems that while my "import everything" script does result in creating __pycache__ files, it doesn't actually have any impact at all on the startup time on a system created from an image with those files. Is there something else that is persisted across python sessions on the same machine other than the .pyc files? Is there a different cache of .pyc files other than that stored in site-packages that might be being checked?