Python import torch takes a long time to init

user_1463767 0
Viewed 769

I've built a system to train neural networks in Python 3.7 using pytorch and when it is ready to start, I run it on an AWS EC2 from an AMI image that is setup to run the training. When I launch a new EC2 instance and run my script it takes a long time > 5 minutes to import torch I've even gone so far as to just start the instance and open a python3 shell and just run import torch and the same thing happens. It is only the first time that I run it that it takes this long. Subsequent runs are fine.

If I press ctrl-c I get

File "/usr/local/lib64/python3.7/site-packages/torch/__init__.py", line 190, in <module>
    from torch._C import *
RuntimeError: KeyboardInterrupt:

When interrupting it, the next time I try it will still take a while to finish.

The EC2 instance is idle while this is happening and there is nothing of note showing in htop

What is causing this to take so long to initialize? Is there anything I can do to speed this up?

Python: 3.7 Torch: 1.7.1+cu110 OS: Amazon Linux 2

torch amazon-web-services python
1 Answers

From cloudonaut's "EBS Snapshot Pitfalls":

Restoring a volume based on an EBS snapshot typically takes a few seconds only. However, the data is not available from the beginning. Instead, EBS restores the data asynchronously. You will notice high latencies during that period.

Consider using AWS's Fast Snapshot feature for ~$500/month, or see alternatives like those described here.

user_52177920
Related