I need to run certain commands after building a Cython module with python setup.py bdist_wheel or an equivalent command. Is this possible?

Specifically here is the situation: I am working on a Cython program for osx. It links to certain .dylib files in my home directory and to get the program to build, you either have to run

export DYLD_LIBRARY_PATH=/path/to/parent/dir

(which is a hassle) or include a directive in setup.py to run

install_name_tool -change @executable_path/my.dylib /path/to/my.dylib

What this basically does is it tells the binary where to look for the dylib. Neither option is ideal, but the latter seems slightly more portable. Right now I just added these instructions to the bottom of setup.py, but this is still not portable: let's say I package and upload my package as follows:

python setup.py sdist  # I think this collects source files into a .tar
python setup.py bdist_wheel  # I think this builds a binary. This is where I assume the post-compilation stuff should happen.
twine upload dist/*   # This puts stuff up on PyPi.

Now, I have read the documentation and a few tutorials, and I am admittedly still not 100% sure what these commands do. But I know that if subsequently, in another project, I run

pip install mypackage

The dylib issue rears its ugly head:

ImportError: dlopen(/path/to/my/module.cpython-36m-darwin.so, 2): 
Library not loaded: @executable_path/my.dylib
  Referenced from: /path/to/my/module.cpython-36m-darwin.so
  Reason: image not found

The solution it seems is to add some kind of automatic post-compilation direction to setup.py to indicate that the install_name_tool operation needs to be performed. Assuming this is possible, how would one do it? Ideally I would like to use off-the-shelf Cython.Build.cythonize and setuptools.Extension, but maybe I need to do something more customized. Thanks for your suggestions!

Solution

Danny correctly pointed out that running install_name_tool is not a portable solution and instead the dylib should somehow be included in the package. delocate is the tool for the job, but the first challenge that I encountered was that delocate doesn't work with @executable_path:

❯ delocate-wheel -w fixed_wheels -v dist/*.whl
Fixing: dist/my-package-1.0.9-cp36-cp36m-macosx_10_12_x86_64.whl
/Users/ethan/virtualenvs/my-package/lib/python3.6/site-packages/delocate/delocating.py:71: UserWarning: Not processing required path @executable_path/my.dylib because it begins with @
  'begins with @'.format(required))

Using otool, I was able to verify that the id of my.dylib uses @executable_path:

❯ otool -L my.dylib
my.dylib:
    @executable_path/my.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
    /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1258.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

To get rid of @executable_path I ran the following:

❯ install_name_tool -id my.dylib my.dylib

Now look at the output of otool -L:

❯ otool -L my.dylib
my.dylib:
    my.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
    /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1258.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

In general, this is not a good id to use for a dylib, but it is enough for delocate-wheel to work with. I rebuilt my .whl:

python setup.py bdist_wheel

And now when I ran delocate_wheel:

❯ delocate-wheel -w fixed_wheels -v dist/*.whl
Fixing: dist/my.whl
Traceback (most recent call last):
...    
delocate.delocating.DelocationError: library "/Users/ethan/my-package/my.dylib" does not exist

This clearly tells us where delocate expects to find my.dylib. So I copied my.dylib to /Users/ethan/my-package/my.dylib and reran the command:

❯ delocate-wheel -w fixed_wheels -v dist/*.whl
Fixing: dist/my.whl
Copied to package .dylibs directory:
  /Users/ethan/my-package/my.dylib

Success! What is this .dylibs directory? I ran tar -xvf my.whl to unpack the wheel and examine its contents:

❯ tree -a fixed_wheels/my-package
fixed_wheels/my-package
├── .dylibs
│   └── my.dylib
├── __init__.py
└── package.cpython-36m-darwin.so

As you can see, the my.dylib has been copied into a .dylibs/ directory that gets packaged in the my.whl. After uploading my.whl to pypi, I was able to download and run the code just fine.