Run makefile from setup.py

Viewed 103

I have a package that uses c++ code. The c++ has a makefile and the python code in the package imports from a .so that is created by the makefile. I succeeded in running the makefile on pip install but the .so file does not seem to be created (it is not found in the directory in site-packages) and at the pip install myModule raise a SyntaxError to where I tried to import the .so file.

The package is structured as the following:

/myModule
├── /src
│   └── /myModule
│       ├── __init__.py
│       ├── main.py
│       └── /cppCode
│           ├── __init__.py
│           ├── wrapper.py
│           ├── C.cpp
│           └── Makefile-conda
│           └── Makefile-gpu
├── LICENSE
├── requirements.txt
├── README.md
└── setup.py

The .so file is called _C.so and it is imported from wrapper.py. This is the error when using pip install:

Installing collected packages: myModule
  *** Error compiling '/Users/user/anaconda3/lib/python3.8/site-packages/myModule/cppCode/wrapper.py'...
    File "/Users/user/anaconda3/lib/python3.8/site-packages/myModule/cppCode/wrapper.py", line 1
      import ._C as c
             ^
  SyntaxError: invalid syntax

setup.py:

import os
import subprocess
from setuptools import setup, find_packages
from setuptools.command.build_ext import build_ext

CONDA_PREFIX = os.environ.get("CONDA_PREFIX", None)
WORKING_DIR = os.getcwd()


class Build(build_ext):
    """Customized setuptools build command - builds protos on build."""
    def run(self):
        run_makefile()
        os.chdir(WORKING_DIR)
        build_ext.run(self)


def makefile_command(gpu: bool):
    if not gpu:
        return ["conda run -n boost make -f Makefile-conda"]
    return ["conda run -n boost make -f Makefile-conda", "conda run -n boost make -f Makefile-gpu"]


def run_makefile():
    # if user use conda run makefile
    if CONDA_PREFIX is not None:
        print("Download version with gpu.")

        # check if gpu exists
        import torch
        if not torch.cuda.is_available():
            print("Does not support GPU.\nBuild cpu.")
            gpu_available = False
        else:
            print("Support GPU.\nBuild GPU.")
            gpu_available = True

        # build Makefile
        os.chdir("src/myModule/cppCode")
        os.system("conda env create -f env.yml --force")
        cmd = '. ' + CONDA_PREFIX + '/etc/profile.d/conda.sh && conda activate boost'
        subprocess.call(cmd, shell=True, executable='/bin/bash')
        for command in makefile_command(gpu_available):
            process = subprocess.Popen(
                command.split(), stdout=subprocess.PIPE)
            output = process.stdout.read()
            print(output)
            output, error = process.communicate()
            print(output)
            print(error)

    else:
        print("Does not use Conda environment or Linux.\nDownload version without gpu")


if __name__ == '__main__':
    # get text for setup
    with open("requirements.txt") as f:
        requirements = [l.strip() for l in f.readlines()]

    with open("README.md") as r:
        readme = r.read()

    setup(
        name="myModule",
        version="0.0.1",
        license="GPL",
        author="Author",
        description_file="README.md",
        license_files="LICENSE.rst",
        install_requires=requirements,
        packages=find_packages('src'),
        python_requires=">=3.6.8",
        include_package_data=True,
        has_ext_modules=lambda: True,
        package_dir={"": "src"},
        cmdclass={
            'build_ext': Build
        },
        easy_install="ok_zip"
    )

Currently, the package can be used only on a Linux machine, because of the use of makefile, which uses Conda because of the use in boost-python, and is installed from pip. I tried to use setuptools.extensions.Extension but I am not really familiar with it, and I am not sure it is the right way because /cppCode has python files in it. Therefore, I am almost sure that the problem is in setup.py.

I use:
python 3.8
Linux
anaconda3
g++



*UPDATE*

The SyntaxError happened due to a wrong relative import. I changed the import in wrapper.py to:

from . import _C as c

Now I do not get any error in the pip install. I wrote test.py which uses wrapper.py and I get the following ImportError:

Traceback (most recent call last):
  File "/Users/user/test.py", line 1, in <module>
    import myModule.cppCode.wrapper as wrap
  File "/Users/user/anaconda3/lib/python3.8/site- 
packages/myModule/cppCode/wrapper.py", line 1, in <module>
    from . import _C as c
ImportError: cannot import name '_C' from 'myModule.cppCode' (/Users/user/anaconda3/lib/python3.8/site-packages/myModule/cppCode/wrapper.py__init__.py)

Although I am not getting an error in pip install, the .so file is still not being created after the pip install.

0 Answers
Related