(Python3) How handle C/C++ dependencies with Pip/Pypi?

Question

I have C/C++ dependencies (TA-Lib) for the Python package that I developed. Not being user-friendly, I don't want the user to install or compile C to use my package. Simply, I just want the user to be able to install my package with pip install .

So, I installed my C dependency in the setup.py before the setup() function by substituing the setup's commands (install, develop, build, etc.).

Here is the code:

import platform
from setuptools import setup
from setuptools.command.install import install
from setuptools.command.develop import develop
from distutils.command.build import build
from distutils.command.build_ext import build_ext
from distutils.command.sdist import sdist
from wheel.bdist_wheel import bdist_wheel
import os

WINDOW_OS = "Windows"
LINUX_OS = "Linux"
CURRENT_OS = platform.system()


def talib_install():
    dir = "build_helper"
    if CURRENT_OS == LINUX_OS:
        os.system(f"cd {dir} && sh talib-install.sh")
    elif CURRENT_OS == WINDOW_OS:
        os.system(f"pip install {dir} TA_Lib-0.4.24-cp39-cp39-win_amd64.whl")
    else:
        raise Exception("Unknown OS")


class TalibBuild(build):
    def get_sub_commands(self):
        # Force "build_ext" invocation.
        commands = build.get_sub_commands(self)
        for c in commands:
            if c == 'build_ext':
                return commands
        return ['build_ext'] + commands


class TalibBuildExt(build_ext):
    def run(self):
        talib_install()
        build_ext.run(self)


class TalibSdit(sdist):
    def run(self) -> None:
        talib_install()
        return super().run()


class TalibDevelop(develop):
    def run(self) -> None:
        talib_install()
        return super().run()

    def finalize_options(self) -> None:
        return super().finalize_options()


class TalibInstall(install):
    def run(self):
        talib_install()
        install.run(self)

    def finalize_options(self) -> None:
        return super().finalize_options()


class TalibBdistWheel(bdist_wheel):
    def run(self):
        talib_install()
        # Run wheel build command
        super().run()

    def finalize_options(self):
        super().finalize_options()


cmdclass = {
    "install": TalibInstall,
    "develop": TalibDevelop,
    "sdist": TalibSdit,
    "bdist_wheel": TalibBdistWheel,
    "build": TalibBuild,
    "build_ext": TalibBuildExt
}

install_requirements = [
    "setuptools==59.5.0",
    "tensorboard",
    "torch-tb-profiler",
    "geneticalgorithm2"
]

talib_install()
setup(
    data_files=["build_helper/talib-install.sh"],
    install_requires=install_requirements,
    cmdclass=cmdclass
)

In local, the following commands are working well: pip install -e . and pip setup.py install.

The problem occurs when I upload my package on Pypi and try installing it on Colab using pip install mypackage. I have the feeling that the installation does not pass through the C code installation step. (I make sure that I have talib-install.sh script in the data_files.)

Moreover, in my release I have two files: wheel archive and source archive. When I execute pip install on Colab, the C code installation works. But pip install doesn't work.

I thought that, by default, pip uses the wheel archive to install. I deleted it so that not having a choice, pip chooses the source. But it doesn't work either...

What is the difference between pip install and pip install ? How pip is working? Any thoughts about how I can resolve my issue?

(I am working on Linux but I want it to work on Windows and MacOs too.)

Thank you very much.

(Python3) How handle C/C++ dependencies with Pip/Pypi?

0 Answers