How to install PyTorch with pipenv and save it to Pipfile and Pipfile.lock?

Viewed 4535

I’m currently using Pipenv to maintain the Python packages used in a specific project. Most of the downloads I’ve tried so far have worked as intended; that is, I enter pipenv install [package] and it installs the package into the virtual environment, then records the package information into both the Pipfile and Pipfile.lock.

However, I’m running into some problems installing PyTorch.

I’ve tried running pipenv install torch, but every time the locking step fails. Instead, I’ve tried forcing a download directly from the PyTorch website using

pipenv run pip install torch===1.6.0 torchvision===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

And it actually installs! If I run pipenv graph it displays both torch and torchvision with their dependencies. But one problem remains: neither torch nor torchvision are being saved into Pipfile and Pipfile.lock.

Any idea on how I can make this happen?

4 Answers

When you use pipenv run pip install <package>, that skips the custom pipenv operations of updating the Pipfile and the Pipfile.lock. It is basically equivalent to doing a plain pip install <package> as if you did not have/use pipenv.

The only way to also update the Pipfile's is to use pipenv install.

Unfortunately, as I'm posting this, pipenv does not have an equivalent for pip's -f/--find-links option. The best solution is to specify pytorch's "https://download.pytorch.org/whl/" URLs as an alternative package index, by adding it as a [[source]] in your Pipfile. See this answer from Mohamad and this answer from Mitch McMabers that describes how to do it. I recommend trying out those answers instead.

A less elegant and quite bad alternative is to manually find the correct torch wheel (.whl) links you need, which usually means looking for the correct link from https://download.pytorch.org/whl/torch_stable.html. Then, create/modify the Pipfile with the specific package versions and URLs to the wheels:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[requires]
python_version = "3.8"

[packages]
torch = {version = "==1.6.0", file = "https://download.pytorch.org/whl/cpu/torch-1.6.0-cp38-none-macosx_10_9_x86_64.whl"}
torchvision = {version = "==0.7.0", file = "https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl"}

Then just do normal pipenv install.

You can confirm the installation with pipenv install --verbose:

Collecting torch==1.6.0
  ...
  Looking up "https://download.pytorch.org/whl/cpu/torch-1.6.0-cp38-none-macosx_10_9_x86_64.whl" in the cache
  Current age based on date: 8
  Starting new HTTPS connection (1): download.pytorch.org:443
  https://download.pytorch.org:443 "GET /whl/cpu/torch-1.6.0-cp38-none-macosx_10_9_x86_64.whl HTTP/1.1" 304 0
  ...
  Added torch==1.6.0 from https://download.pytorch.org/whl/cpu/torch-1.6.0-cp38-none-macosx_10_9_x86_64.whl#egg=torch 
...
Successfully installed torch-1.6.0

Collecting torchvision==0.7.0
  ...
  Looking up "https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl" in the cache
  Current age based on date: 8
  Starting new HTTPS connection (1): download.pytorch.org:443
  https://download.pytorch.org:443 "GET /whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl HTTP/1.1" 304 0
  ...
  Added torchvision==0.7.0 from https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl#egg=torchvision
...
Successfully installed torchvision-0.7.0

This also adds entries to Pipfile.lock:

"torch": {
    "file": "https://download.pytorch.org/whl/cpu/torch-1.6.0-cp38-none-macosx_10_9_x86_64.whl",
    "hashes": [
        ...
    ],
    "index": "pypi",
    "version": "==1.6.0"
},
"torchvision": {
    "file": "https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl",
    "hashes": [
        ...
    ],
    "index": "pypi",
    "version": "==0.7.0"
}

With that, you now have a Pipfile and Pipfile.lock that you can check-in/commit to version control and track/manage as you develop your application.

Instead of manually editing the Pipfile, you can also do it from the command line:

(temp) $ pipenv install --verbose "https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl"
Installing https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl...
...
Adding torchvision to Pipfile's [packages]...
✔ Installation Succeeded

That should also add an entry to the Pipfile:

[packages]
...
torchvision = {file = "https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl"}

Of course, this all depends on finding out which wheel you actually need. This can be done by first using a plain pip install <package> with the -f/--find-links option targeting the https://download.pytorch.org/whl/torch_stable.html URL, then checking which wheel it used.

  1. First, let's get the correct .whl file with pip install
    $ pipenv run pip install --verbose torchvision==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
    Looking in links: https://download.pytorch.org/whl/torch_stable.html
    ...
    Collecting torchvision==0.7.0
      Downloading torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl (387 kB)
    ...
    
  2. Remove pip install-ed things from the virtual environment
    $ pipenv clean
    
  3. Repeat installation but using pipenv install
    $ pipenv install --verbose "https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl"
    
    • Just combine "https://download.pytorch.org/whl/" + .whl filename from step 1

It might seem a bit backwards using pip install first then copying it over to pipenv, but the objective here is to let pipenv update the Pipfile and Pipfile.lock (to support deterministic builds) and to "document" your env for version control.

you can install it by adding the PyTorch source to your pip file as following:

[[source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/"
verify_ssl = true

[packages]
torch = {index = "pytorch",version = "==1.9.0"}
torchvision = {index ="pytorch", version= "==0.10.0"}
torchaudio = {index ="pytorch", version= "==0.9.0"}

[requires]
python_version = "3.7"

And then run pipenv install

Note: The index refers to the source name.

you can do that with any similar cases: ex: Install pytorch-geometric:

[[source]]
name = "pytorch-geometric"
url = "https://pytorch-geometric.com/whl/torch-1.9.0+cu111.html"
verify_ssl = true
[packages]

torch-scatter = {index= "pytorch-geometric", version= "==2.0.7"}
torch-sparse = {index= "pytorch-geometric", version= "==0.6.10"}
torch-cluster = {index= "pytorch-geometric", version= "==1.5.9"}
torch-geometric = {index= "pytorch-geometric", version= "==1.3.2"}

(Notice: The most up-to-date version of this post is always at the Pipenv project site, since I can't maintain the same edits in multiple places forever.)


This technique is the correct one and it's documented here:

https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes

Basically, you add a new "source repository" to your Pipfile. And you should always use verify_ssl = true for extra security when possible.

Anyway, there's a shortcut to adding a custom repository without editing the pipfile. The --extra-index-url shortcut says "Add this PEP 503-standard repo URL":

  • Version that uses CUDA Toolkit 11.3, with GPU acceleration (this toolkit is required by NVIDIA Ampere (RTX 30x0) GPUs and newer):
pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.10.1+cu113"
  • Version that uses CUDA Toolkit 10.2, with GPU acceleration:
pipenv install --extra-index-url https://download.pytorch.org/whl/ "torch==1.10.1+cu102"
  • Version that uses CPU instead:
pipenv install --extra-index-url https://download.pytorch.org/whl/ "torch==1.10.1+cpu"

Now you may notice something: There's no ".html" in my repo filenames.

That's because the repos with a HTML filename (such as https://download.pytorch.org/whl/torch_stable.html) are NON-STANDARD repos that ONLY pip understands.

You CANNOT use those URLs in Pipenv.

Pipenv uses PEP 503 standardized repos. Torch publishes those at the URLs WITHOUT ".html" filenames. So remove the filename to get the proper repo URLs for Pipenv usage.

This fact is documented here, when Torch's team decided to finally publish PEP 503 repos: https://github.com/pytorch/pytorch/issues/25639#issuecomment-861707149

You may notice something else: I always specify a version with +cpu or +cu113 or whatever.

This is NECESSARY, because otherwise Pipenv doesn't know which version to install and won't get GPU acceleration. The "architecture" (GPU vs CPU vs different CUDA toolkits) is baked into the version number. It's literally what Torch decided to do, and the ONLY correct way to download them is to specify the exact version and architecture (such as +cu113) you want.

If you want to figure out the URL to the correct repos/versions in the future, do as follows:

Go to pytorch.org and select "Stable, Pip, Python, CUDA 11.3" (or whatever is the latest CUDA you may be using), and then the command textbox will reveal the repo URL for the latest CUDA toolkit. You MUST then remove the HTML filename because that's the pip-specific repo.

Visit the resulting URL, such as https://download.pytorch.org/whl/cu113/, and navigate into the desired library folder. Then search for the version you wanted, and be sure to specify its FULL version specifier, such as 1.10.1+cu113.

Oh and sometimes, the Torch team forgets to upload the latest version to the PEP 503 repo, such as right now, where 1.10.2 is available but only in the non-standard pip-style repo. That's why you should always visit the PEP 503 repo to see the latest version they've uploaded into that repo. (I would love if they completely stop providing the old-school pip-repo at all, since pip supports PEP 503 too, so they should focus on the standards-compliant repo...)

See my examples above for perfect installation instructions for the current versions as of this writing.

One final note: ALWAYS RESPECT THE TORCH PYTHON VERSIONS.

The packages will have names such as "cp37", "cp38", "cp39". As of this writing, the highest version they have created is for Python 3.9. There's no Python 3.10 version of Torch.

Therefore, it helps to install Pyenv and specify an exact Python version in your Pipenv via pipenv install --python=3.9 to ensure that you have the latest version that Torch supports and not anything "too new/unsupported". :)

Gino Mempin's solution worked for me (I only had to change the url slightly to "https://download.pytorch.org/whl/cpu/", so add "cpu/"). Then I tried the following and it worked as well and is easier imo and you don't need to specify the version.

Pipfile:

...
[[source]]
url = "https://download.pytorch.org/whl/cpu/"
verify_ssl = false
name = "pytorch"

[packages]
torch = {index = "pytorch", version = "*"}
...

Then just pipenv update.

This is for the cpu version of pytorch; there is another folder "https://download.pytorch.org/whl/cu80/" which I assume would work similarly for the cuda version.

Related