Verifying the integrity of PyPI Python packages

user_2514936 0
Viewed 4184

Recently there came some news about some Malicious Libraries that were uploaded into Python Package Index (PyPI), see:

  1. Malicious libraries on PyPI
  2. Malicious modules found into official Python repository (this link contains the list of malicious packages)
  3. Developers using malicious Python Modules

I am not trying to forward these news but I am trying to prevent myself and other teammates to identify if a package from PyPI has not been altered by an external party.

Questions:

  1. What security check should I use once I have downloaded a package from PyPI? MD5 or any extra step?
  2. Is MD5 signature enough to verify the integrity of Python Packages?
python-2.7 pypi python-3.x security python
2 Answers

First, your concern of obtaining malicious files when downloading from PyPI using pip is valid. In fact as of 2020, pip has no way to cryptographically validate the authenticity and integrity of the software it downloads (but it has been a TODO item since 2013)

Second, md5 can verify the integrity of a file, but it can't do so securely. Nor is checking the integrity sufficient. To securely verify that a file downloaded from pip is the file you expect, you have to:

  1. Authenticate the the file by confirming that it was in-fact produced by the publisher (using a cryptographic signature) and
  2. Verify the integrity of the file after download

Though pip doesn't have built-in support for this, you can do it manually by downloading the file you want and its detached signature file from PyPI manually, followed by checking the signature using gpg.

For example, the borgbackup project can be viewed here on PyPI's website:

Clicking the "Download Files" button gives you an option to download the lastest tarball at the following URL:

Alternatively, you can also get this URL using cURL against the PyPI "simple" API

user@disp5066:~$ curl -s https://pypi.org/simple/borgbackup/ | grep -i borgbackup-1.1.13.tar.gz
    <a href="https://files.pythonhosted.org/packages/97/68/27d96a12f54894223ad6676ce4d215ad61771e3e723580f3ee6e609e17b7/borgbackup-1.1.13.tar.gz#sha256=164a8666a61071ce2fa6c60627c7646f12e3a8e74cd38f046be72f5ea91b3821">borgbackup-1.1.13.tar.gz</a><br/>
user@disp5066:~$

To get the signature of this file, simply append .asc to the URL:

user@disp5066:~$ wget https://files.pythonhosted.org/packages/97/68/27d96a12f54894223ad6676ce4d215ad61771e3e723580f3ee6e609e17b7/borgbackup-1.1.13.tar.gz.asc
--2020-07-02 07:51:12--  https://files.pythonhosted.org/packages/97/68/27d96a12f54894223ad6676ce4d215ad61771e3e723580f3ee6e609e17b7/borgbackup-1.1.13.tar.gz.asc
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.37.63, 2a04:4e42:9::319
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.37.63|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 862 [application/octet-stream]
Saving to: ‘borgbackup-1.1.13.tar.gz.asc’

borgbackup-1.1.13.t 100%[===================>]     862  --.-KB/s    in 0s      

2020-07-02 07:51:14 (37.2 MB/s) - ‘borgbackup-1.1.13.tar.gz.asc’ saved [862/862]

user@disp5066:~$ cat borgbackup-1.1.13.tar.gz.asc 
-----BEGIN PGP SIGNATURE-----

iQJHBAABCgAxFiEEL4Gv+6sE4R/o7mXUJDrPqVH3jgEFAl7cGqwTHHR3QHdhbGRt
YW5uLWVkdi5kZQAKCRAkOs+pUfeOAd9ND/4nm2O7CK5a4aK41jAI1NisbgEtEJup
SiD6bvMKpo3VU0P/3Y6pUibKOGzaRImBTB04qS3LlgjB0mCp1RSVsj/Hn+yCNw+k
hfUH7E7JgAkq96Vkv1dcYgaJ9nhzuIAkEf0aDyzSo8HkBvGGN0/tfCQ7Nr7hI21u
v5qupIyu7KZrBwY389l7+6yJ9G5qCtHU0fDALRYyjsX+WphrAaizrhFZJO7Km8VZ
gZhAz3WUDPFwgNMb1mToUxpI2ZpnYnRxVBwjnX0Ps77ua4F5OsYM+hYwH5eX9bS9
gmb+W3NjUNjVVj4z+OgN8FGbCTeFVQ6E+IVdm55D4ZRU8KarvFoKOI7HS4GP/3iv
4iWqDaYBMRShnUTk1FKFCKjTb5tXewUGPwio+4bpgUyfJj0OWj1ecMqeF5VAslWz
6pZnsUqLpTFuHUA6dr18TKX4U+c6rdXVM7BhNZe2XtjaQwau6Wz9nC1xhZyFNl1q
CHY7jmLhsfP8GXkh31X9bJrKSZMyYRYat2e7kOroIJczRcHG9T708T+KzsfAb+6w
pWZbfWNfCbCmVQehyhDvNepB3IB5w6ijrZwKTamHAnYBVkAUD/aYwDQJf4nAL4YI
7JXBRpLlCVQGRUQdClqy8QjzpSZs5/Dbetvy5of753JbVjFQtGO2gLLp0wL0HB0v
vIZv3dfBDvfcXQ==
=F4gj
-----END PGP SIGNATURE-----
user@disp5066:~$

Pip also has no way to mark the gpg key used for signing a package as the "official" key, so you have to hope that the author has published their key in a few distinct domains so you can fetch it out-of-band a few ways to confirm it's correct then import it and trust it. After that, you can use gpg to verify the signature of the file downloaded from PyPI.

gpg --verify borgbackup-1.1.13.tar.gz.asc

See also:

  1. https://github.com/borgbackup/borg/issues/4213
  2. https://security.stackexchange.com/questions/232855/does-pythons-pip-provide-cryptographic-authentication-and-integrity-validation
user_11741020
Related