How do I have multiple versions of numpy installed on Amazon EMR and how to I delete the early versions?

Question

I do not understand how Python can have multiple versions of a single package installed, or why, when I have multiple versions installed, import package does not give me the most recent one.

I am using AWS linux and the AWS repo in AWS EMR. When I install Python 3.6 it gives me Numpy version 1.14.5 by default. I want to use version 1.19.1 and have put it in a directory.

$ sudo pip-3.6 install --upgrade --no-index --find-links /mnt/packages/ numpy==1.19.1
Requirement already up-to-date: numpy==1.19.1 in /usr/local/lib64/python3.6/site-packages
$

But notice:

 $ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.version.version
'1.14.5'
>>>

Because I have both version 1.14.5 and version 1.19.1 installed.

I can uninstall version 1.19.1 and then I can uninstall version 1.14.5, although it seems that I have to uninstall it twice (see below for the extended remix).

So how do I make it so that import numpy gets version 1.19.1 and not the older version?

$ sudo pip-3.6 uninstall numpy
Uninstalling numpy-1.14.5:
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.data/scripts/f2py
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/METADATA
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/RECORD
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/WHEEL
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/top_level.txt
  /usr/local/lib64/python3.6/site-packages/numpy/.libs/libgfortran-ed201abd.so.3.0.0
  /usr/local/lib64/python3.6/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so
  /usr/local/lib64/python3.6/site-packages/numpy/_import_tools.py
  /usr/local/lib64/python3.6/site-packages/numpy/add_newdocs.py
  /usr/local/lib64/python3.6/site-packages/numpy/core/_dummy.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/multiarray_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/operand_flag_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/struct_ufunc_test.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/test_rational.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/umath_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/__version__.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/compat.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/environment.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/f2py/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/fftpack.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/fft/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/tests/test_fftpack.py
  /usr/local/lib64/python3.6/site-packages/numpy/lib/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/linalg/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/ma/version.py
  /usr/local/lib64/python3.6/site-packages/numpy/random/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/random/randomkit.h
  /usr/local/lib64/python3.6/site-packages/numpy/testing/decorators.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/__init__.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/decorators.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/noseclasses.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/nosetester.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/parameterized.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/utils.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/noseclasses.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nosetester.py
Proceed (y/n)? y
  Successfully uninstalled numpy-1.14.5
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>
$ sudo pip-3.6 uninstall numpy
Uninstalling numpy-1.14.5:
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.data/scripts/f2py
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/METADATA
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/RECORD
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/WHEEL
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/top_level.txt
  /usr/local/lib/python3.6/site-packages/numpy/.libs/libgfortran-ed201abd.so.3.0.0
  /usr/local/lib/python3.6/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so
  /usr/local/lib/python3.6/site-packages/numpy/LICENSE.txt
  /usr/local/lib/python3.6/site-packages/numpy/__config__.py
  /usr/local/lib/python3.6/site-packages/numpy/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/_distributor_init.py
  /usr/local/lib/python3.6/site-packages/numpy/_globals.py
  /usr/local/lib/python3.6/site-packages/numpy/_import_tools.py
  /usr/local/lib/python3.6/site-packages/numpy/add_newdocs.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/_inspect.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/py3k.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/conftest.py
  /usr/local/lib/python3.6/site-packages/numpy/core/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/core/_dummy.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib/python3.6/site-packages/numpy/core/_internal.py
  /usr/local/lib/python3.6/site-packages/numpy/core/_methods.py
  /usr/local/lib/python3.6/site-packages/numpy/core/arrayprint.py
  /usr/local/lib/python3.6/site-packages/numpy/core/cversions.py
  /usr/local/lib/python3.6/site-packages/numpy/core/defchararray.py
  /usr/local/lib/python3.6/site-packages/numpy/core/einsumfunc.py
  /usr/local/lib/python3.6/site-packages/numpy/core/fromnumeric.py
  /usr/local/lib/python3.6/site-packages/numpy/core/function_base.py
  /usr/local/lib/python3.6/site-packages/numpy/core/generate_numpy_api.py
  /usr/local/lib/python3.6/site-packages/numpy/core/getlimits.py
  /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/__multiarray_api.h
  /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/__ufunc_api.h
...
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/intel.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/lahey.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/mips.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/nag.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/none.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/pathf95.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/pg.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/sun.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/vast.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/from_template.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/intelccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/lib2def.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/line_endings.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/log.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/mingw/gfortran_vs2003_hack.c
  /usr/local/lib/python3.6/site-packages/numpy/distutils/mingw32ccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/misc_util.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/msvc9compiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/msvccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/npy_pkg_config.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/numpy_distribution.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/pathccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/system_info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_exec_command.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_gnu.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_intel.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_nagfor.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_misc_util.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_npy_pkg_config.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_system_info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/unixccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/basics.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/broadcasting.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/byteswapping.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/constants.py
...
  /usr/local/lib/python3.6/site-packages/numpy/testing/print_coercion_tables.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_decorators.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_doctesting.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_utils.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/utils.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_ctypeslib.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_matlib.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_numpy_version.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_reloading.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_scripts.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_warnings.py
  /usr/local/lib/python3.6/site-packages/numpy/version.py
Proceed (y/n)? Y
  Successfully uninstalled numpy-1.14.5
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ sudo pip-3.6 uninstall numpy
Cannot uninstall requirement numpy, not installed
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ sudo pip-3.6 uninstall numpy
Cannot uninstall requirement numpy, not installed
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "", line 1, in 
ModuleNotFoundError: No module named 'numpy'
>>>
$ sudo pip-3.6 install --upgrade --no-index --find-links /mnt/gits/das-vm-config/dist/python numpy==1.19.1
Collecting numpy==1.19.1
Installing collected packages: numpy
Successfully installed numpy-1.19.1
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.version.version
'1.19.1'
>>>

user_51167 · Answer

Our solution is to do the following in the EMR bootstrap script (which gets run on every node):

Use yum to install python36 and python36-pip
use pip-3.6 with the --target option to install numpy-1.19.1 in /mnt/python_additions. We do this in the Bootstrap action so that it happens on every node.
Let Amazon's script install python36-numpy, which installs numpy-1.14.4 in the site library location.
In the program that gets called from spark-submit, add /mnt/python_additions to sys.path.

It's a hack, but it works.

user_8071560 · Answer

I had just run into the same issue on my end. I ended up running the following 3 times to uninstall all numpy versions:

python3 -m pip uninstall numpy

Once it told me there is no numpy installed, I ran the install and it was able to give me the proper version.

Quite annoying, and wasn't an issue before.

user_2643556 · Answer

Cross posting the answer which worked for me from another SO question: https://stackoverflow.com/a/69367430/2643556

user_1753947 · Answer

Whilst you can have multiple versions of a package installed, only one is importable in a Python environment. If you want to be explicit about what versions of a package you want to use, So see where Python is importing packages from, you can run the following in an interactive session:

>>> import sys
>>> print(sys.path)

This will return a list of import locations. The last item in the list show the last place Python will look. Later paths in the list superceed, earlier ones. If you had the following:

["site_packages_path_with_numpy_1.19.1", "site_packages_with_numpy_1.14.5"]

Python would import numpy 1.14.5.

If you've not come across them, I suggest looking into virtualenv and requirements.txt.

Virtualenv allows you to create a local version of Python (separate from the system one) with whatever packages you want. Eg.

python -m venv .venv
source .venv/bin/activate

This will create a new copy of Python and actives it. Calls to Python or PIP will now use the virtual environment.

You can install packages into the virtual environment like so.

pip install numpy==1.19.1

Requirements allows you to keep a list of what packages you'd like to install. The file looks like so

# requirements.txt

numpy==1.19.1

You can install the contents of requirements into the virtual environment like so (assuming you've run the active command):

pip install -r requirements.txt

Finally, a more modern and arguably better solution is https://python-poetry.org/.

How do I have multiple versions of numpy installed on Amazon EMR and how to I delete the early versions?

4 Answers