Is it a good practice to upgrade all python packages in production to their latest versions?

The common pattern for versioning python modules (and many other software) is major.minor.patch where after the initial release, patch versions don't change the api, minor releases may change the api in a backward compatible way, and major releases usually aren't backward compatible

so if you have module==x.y.z a relatively safe requirement specification would be:

module>=x.y.z,<x.y+1.0

note that while this will usually be OK, it's based on common practices, and not guaranteed to work, and it's more likely to be stable with more "organized" modules

Do not update packages in production, you can brake things, if you have a package which has tables in database, and you update it you can brake your database. I used for example python social auth, I wanted to upgrade it to the last version, so for that, I need it to upgrade do version x, run migrations and after that got to last version and migrate. Upgrade the packages in your development environment, test it. If you have a pre-production, do that there after testing in dev.

One of the most robust practices in the industry is to ensure that your code is well unit-tested, well property-tested, and well regression-tested. Once you've got good coverage and the tests are green, begin upgrading your dependencies. You might want to do this one at a time if you're feeling ambitious, or you might give it all a go and try your luck. If after upgrading your tests are still green and you can manually go through all workflows, chances are you're in the clear! Those tests you spent time on can then be leveraged going forward for any smaller, incremental upgrades.

First, a little background on me. I'm a security test engineer and work heavily on Python, that being said most of my work is test code and the products I typically work on do not ship with Python running on the box so I cannot give you direct answers to questions about specific Python packages. I can give you some rules of thumb about what most companies do when faced with such questions in a general sense.

Figure out if you need to update:

1. Run a security scanner (like Nessus). If you have any glaring version problems with your application stack or hosts Nessus will give you some idea of things that should be fixed immediately

2. Check the release notes for all of the packages to see if anything actually needs updating. Pay extra close attention to any package that opens a socket. If there are no security fixes it probably isn't worth your time to update. In some cases even if there is a security problem in a package, it may be something you aren't using so pay attention to the problem description. If you have an application in production now, typically the goal is to change as little as possible to limit the amount of code update and testing that needs to be done.

If you've discovered that you need an update to start resolving dependencies:

1. Probably the easiest way to do this is to set up a test environment and start installing the updated versions that you need. Once more, try to make a few changes as possible while installing your code in the test environment. In Python, there is a reliance for C libraries for much of the security stuff, In your test environment be sure to have the same lower level system libs as well...

2. Create a list of changes that need to be made and start devising a test plan for the sections of code that you need to replace.

Once your development environment is set up, test test test. You'll probably want to rerun your security scanner and be assured to exercise all of the new code.

It depends. Most of the time, updating the packages for minor version changes (eg: 2.45.1 -> 2.56.1) is unlikely to break your system. However, it is still advisable to run extensive regression testing. However, major version changes(eg. 2.45.1 to 3.13.0) should definitely be avoided as most of them have little backward compatibility. An example of this would be the selenium web driver which in version 3.0 cannot run without geckodriver compared to version 2.56. Regardless, extensive unit and regressive testing should be carried out on the old and new code to ensure there are no unexpected changes, especially at the corner cases.

Considering that you have mentioned that you have mentioned that you only use a single python pip, I can say that this will be an issue for future programs as you may need to use different versions of the same package for different projects.

An acceptable practice to avoid such issues would be to use a virtual environment for each project. Just create a makefile in your project root and use: virtualenv --python=${PYTHON} env env/bin/pip install --upgrade pip env/bin/pip --upgrade setuptools wheel pip-tools

Then have env/bin/pip install the requirements from your requirements.txt file.

You are absolutely right. There will be backward incompatibility issue. Do not update packages to the latest blindly. Most likely, you will have package/module/class/variable/key undefined/notFound issues. Especially you have a complex system. Even if you use pip install --upgrade somepackage

This is lesson from my real experience.