pip: downloading dependencies to specific platform including non binaries

user_8590282 0
Viewed 4548

I'm trying to download the dependencies of paramiko from a linux host to a windows target which has no internet access .

After reading the example on pip's documentation I've used to following command in order to download the dependencies recursively to a 64 bit windows platform:

pip3 download --only-binary=:all: --platform win_amd64 --implementation cp paramiko

Was able to recursively download the dependencies until reaching pycparser. That is not surprising since I've used the --only-binary=:all: flag. Thing is - pip forces the usage of this flag when --platform flag is passed:

ERROR: --only-binary=:all: must be set and --no-binary must not be set (or must be set to :none:) when restricting platform and interpreter constraints using --python-version, --platform, --abi, or --implementation.

Terminal produced the following output:

Collecting paramiko
  Downloading paramiko-2.3.0-py2.py3-none-any.whl (182kB)
    100% |████████████████████████████████| 184kB 340kB/s 
  Saved ./paramiko-2.3.0-py2.py3-none-any.whl
Collecting pynacl>=1.0.1 (from paramiko)
  Using cached PyNaCl-1.1.2-cp35-cp35m-win_amd64.whl
  Saved ./PyNaCl-1.1.2-cp35-cp35m-win_amd64.whl
Collecting cryptography>=1.5 (from paramiko)
  Using cached cryptography-2.0.3-cp35-cp35m-win_amd64.whl
  Saved ./cryptography-2.0.3-cp35-cp35m-win_amd64.whl
Collecting pyasn1>=0.1.7 (from paramiko)
  Using cached pyasn1-0.3.5-py2.py3-none-any.whl
  Saved ./pyasn1-0.3.5-py2.py3-none-any.whl
Collecting bcrypt>=3.1.3 (from paramiko)
  Using cached bcrypt-3.1.3-cp35-cp35m-win_amd64.whl
  Saved ./bcrypt-3.1.3-cp35-cp35m-win_amd64.whl
Collecting cffi>=1.4.1 (from pynacl>=1.0.1->paramiko)
  Using cached cffi-1.11.0-cp35-cp35m-win_amd64.whl
  Saved ./cffi-1.11.0-cp35-cp35m-win_amd64.whl
Collecting six (from pynacl>=1.0.1->paramiko)
  Using cached six-1.11.0-py2.py3-none-any.whl
  Saved ./six-1.11.0-py2.py3-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.5->paramiko)
  Using cached asn1crypto-0.22.0-py2.py3-none-any.whl
  Saved ./asn1crypto-0.22.0-py2.py3-none-any.whl
Collecting idna>=2.1 (from cryptography>=1.5->paramiko)
  Using cached idna-2.6-py2.py3-none-any.whl
  Saved ./idna-2.6-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.4.1->pynacl>=1.0.1->paramiko)
  Could not find a version that satisfies the requirement pycparser (from cffi>=1.4.1->pynacl>=1.0.1->paramiko) (from versions: )
No matching distribution found for pycparser (from cffi>=1.4.1->pynacl>=1.0.1->paramiko)

Is there a way of overcoming this issue? Will I have to manually install non-binary packages (and their dependencies)?

Thanks, Joey.

pip installation cross-platform python
4 Answers

You have two options:

  1. run the download operation on the same platform (be careful to be the same)
  2. fix the internet access on your host

Don't try other fancy method or you will shut yourself in the foot: some dependencies will need to compile!

user_998340

You can use the --prefer-binary option in pip. That'll make pip consider wheels as more important, even if they're an older version than an existing sdist (sdist is short for source distribution). An sdist would be selected if no wheels are found to be compatible.

This was released in pip 18.0 (so that's early 2018, pip's using CalVer now).

user_19312740

@sorin is right, your only real option is to use the exact same environment to download the dependencies as you'll be installing them on.

My solution is to use Docker to build a wheel that matches the target platform. In my case it is Debian 10, but it will work just the same for any operating system and version as long as there is a Docker image available.

Example Dockerfile to build a wheel with the dependencies in requirements.txt for Debian 10 with cPython 3.9:

FROM python:3.9-slim-buster

COPY requirements.txt requirements.txt

RUN set -eux; \
    apt-get update && \
    apt-get install -y build-essential && \
    python3 -m venv .venv --without-pip

ENV VIRTUAL_ENV=.venv
ENV PATH="${VIRTUAL_ENV}/bin:${PATH}"

RUN set -eux; \
    curl --silent https://bootstrap.pypa.io/get-pip.py | python && \
    pip download --prefer-binary --upgrade setuptools wheel setuptools-rust -d deps && \
    pip download --prefer-binary -r requirements.txt -d deps && \
    mkdir -p main-wheel && \
    pip wheel --wheel-dir=main-wheel -r requirements.txt

Build the image and extract the wheel:

docker build -t buildwheel -f Dockerfile
mkdir -p artifacts
CONTAINER=$(docker create buildwheel || exit 1)
docker cp "${CONTAINER}":main-wheel artifacts/. || exit 1
docker rm "${CONTAINER}"
docker image rm buildwheel

Congrats, you now have a wheel specifically for Debian 10 with cPython 3.9 inside the directory artifacts/main-wheel. Copy it to the target machine and do pip install --no-index --find-links=artifacts/main-wheel -r requirements.txt and everything should work.

PS: You might need to add build-time dependencies to the apt-get install inside the Dockerfile.

user_6159870

in python3 you can download the dependencies like mentioned below

while running this be inside the folder you want to save

pip download -r requirements.txt

once you downloaded the files move these to the machine you want to you can install

then run this command

pip install -r req.txt --no-index --find-links="/path/to/downloaded/files"
user_123539490
Related