Two pip-installed modules have the same name, how to select which one is loaded?

Viewed 986

I'm writing a python program that relies on a specific module, rtmidi. Thing is, at least two different packages in PyPI have a module with that name, rtmidi and python-rtmidi. They offer almost the same functionalities but have different syntax.

When only the "right" package is installed everything works fine. But if both packages are installed, using import rtmidi loads the "wrong" module and my program crashes. The only way to get it working again is to uninstall both packages then re-install the right one. Of course, since the user might rely on the other module for other programs, I can't expect them to do that.

Trying to identify the module with rtmidi.__name__ gives the same result with both packages.

So my question, how do I go about resolving this name clash problem? Is there a best-practice way to handle this?

4 Answers

You can't have them both installed (if relying on default pip behavior). Here I'll demonstrate with a couple simple pure-Python packages that have an import-name conflict: coveralls and python-coveralls.

# in a fresh environment
pip install python-coveralls
python -c "import coveralls; print(dir(coveralls)); print(coveralls.__file__)"
# ['__author__', '__builtins__', '__cached__', '__classifiers__', '__copyright__', 
# '__doc__', '__docformat__', '__file__', '__license__', '__loader__', '__name__', 
# '__package__', '__path__', '__spec__', '__version__', 'parse_args', 'wear']
# /path/to/lib/python3.8/site-packages/coveralls/__init__.py

These are the contents of python-coveralls. Note that the actual __init__.py file is located in the site-packages/coveralls/ directory.

pip install coveralls
python -c "import coveralls; print(dir(coveralls)); print(coveralls.__file__)"
# ['Coveralls', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', 
# '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', 
# 'api', 'exception', 'git', 'reporter', 'version']
# /path/to/lib/python3.8/site-packages/coveralls/__init__.py

These are the contents of coveralls. python-coveralls has been overwritten. Note that this is the same file. Anything in the old __init__.py is gone.

pip uninstall coveralls
python -c "import coveralls; print(dir(coveralls)); print(coveralls.__file__)"
# ['__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', 
# '__spec__']
# None

There's still a ghost of the package there, but its contents are gone (except things that all packages have internally). Notice that the file is now None: there is no __init__.py file for this package.

You can un-bork your environment by also running pip uninstall python-coveralls and then reinstalling whichever you want.

Solution

You do have to require that your users only use the package that is in your requirements, but that's why we use virtual environments. Alternatively, a user that wants to directly use the other package can change the install location (and thus the name used when loading) with the --target option to pip (but that won't help other apps that use the other library).

In practice, it's probably best to think of this as part of your installation process. You either need to (a) tell your users what requirements they need to install; or (b) have some automated process that gets the right requirements.

I'd say best practice is (b). A common way of doing this is to use setuptools, and defining the install_requires argument to the setup function. This refers to the package name on PyPI, not the import name. The setuptools quickstart is a good place to start.

The following is based on the comments by hoefling and the answer by dwhswenson (the latter explains well what the problem is).

  1. Flag it to the project owners so they change the names of their modules. In theory this is by far the best solution as not only it will solve your problem but also save others from it, which is in the best interest of the module's devs. In practice it opens a whole can of worms as to which one should change name.

  2. Or include the module in your project, aka vendoring, and import it using the path my_project/_vendor/the_module (or add that path first in sys.path). If the module is just a .py file and the license permits it, this is the simplest way to resolve the issue. This wasn't doable in my case because the module I needed includes some C code that needs to be compiled, and including the whole source code then requiring the user to compile it seemed like big ask. So I had to do this:

  3. Or tell the user to install the module in your project's folder. Using pip with the option --target followed by a folder path inside your project, you can get your own copy of the module which won't overwrite the system one. You can then check if that folder exists before importing. If it does, import the module from there, if not you can check if the system-installed module is the right one, and prompt the user if it isn't.

And here's a rough example of how I implemented the code in my project. It replaces import rtmidi

if os.path.isdir('my_project/rtmidi'):
    from .rtmidi.rtmidi import _rtmidi as rtmidi
else:
    try:
        import rtmidi
    except ImportError:
        print('Please run: pip3 install rtmidi')
        exit()
    try:
        # something that only works with the right module:
        test = rtmidi.MidiMessage() 
    except:
        print('Please run: pip3 install --target=./my_project/rtmidi rtmidi')
        exit()

Note: I should probably just try to import the local folder module instead of using a conditional statement, and use the path my_project/_vendor/rtmidi.

i think you should see which one you dont want and delete it then try importing it again. you can find packages in site-packages folder.

If package names are different import it with package name if are the same you can change package name

Related