I have a package that uses c++ code. The c++ has a makefile and the python code in the package imports from a .so that is created by the makefile. I succeeded in running the makefile on pip install but the .so file does not seem to be created (it is not found in the directory in site-packages) and at the pip install myModule raise a SyntaxError to where I tried to import the .so file.
The package is structured as the following:
/myModule
├── /src
│ └── /myModule
│ ├── __init__.py
│ ├── main.py
│ └── /cppCode
│ ├── __init__.py
│ ├── wrapper.py
│ ├── C.cpp
│ └── Makefile-conda
│ └── Makefile-gpu
├── LICENSE
├── requirements.txt
├── README.md
└── setup.py
The .so file is called _C.so and it is imported from wrapper.py.
This is the error when using pip install:
Installing collected packages: myModule
*** Error compiling '/Users/user/anaconda3/lib/python3.8/site-packages/myModule/cppCode/wrapper.py'...
File "/Users/user/anaconda3/lib/python3.8/site-packages/myModule/cppCode/wrapper.py", line 1
import ._C as c
^
SyntaxError: invalid syntax
setup.py:
import os
import subprocess
from setuptools import setup, find_packages
from setuptools.command.build_ext import build_ext
CONDA_PREFIX = os.environ.get("CONDA_PREFIX", None)
WORKING_DIR = os.getcwd()
class Build(build_ext):
"""Customized setuptools build command - builds protos on build."""
def run(self):
run_makefile()
os.chdir(WORKING_DIR)
build_ext.run(self)
def makefile_command(gpu: bool):
if not gpu:
return ["conda run -n boost make -f Makefile-conda"]
return ["conda run -n boost make -f Makefile-conda", "conda run -n boost make -f Makefile-gpu"]
def run_makefile():
# if user use conda run makefile
if CONDA_PREFIX is not None:
print("Download version with gpu.")
# check if gpu exists
import torch
if not torch.cuda.is_available():
print("Does not support GPU.\nBuild cpu.")
gpu_available = False
else:
print("Support GPU.\nBuild GPU.")
gpu_available = True
# build Makefile
os.chdir("src/myModule/cppCode")
os.system("conda env create -f env.yml --force")
cmd = '. ' + CONDA_PREFIX + '/etc/profile.d/conda.sh && conda activate boost'
subprocess.call(cmd, shell=True, executable='/bin/bash')
for command in makefile_command(gpu_available):
process = subprocess.Popen(
command.split(), stdout=subprocess.PIPE)
output = process.stdout.read()
print(output)
output, error = process.communicate()
print(output)
print(error)
else:
print("Does not use Conda environment or Linux.\nDownload version without gpu")
if __name__ == '__main__':
# get text for setup
with open("requirements.txt") as f:
requirements = [l.strip() for l in f.readlines()]
with open("README.md") as r:
readme = r.read()
setup(
name="myModule",
version="0.0.1",
license="GPL",
author="Author",
description_file="README.md",
license_files="LICENSE.rst",
install_requires=requirements,
packages=find_packages('src'),
python_requires=">=3.6.8",
include_package_data=True,
has_ext_modules=lambda: True,
package_dir={"": "src"},
cmdclass={
'build_ext': Build
},
easy_install="ok_zip"
)
Currently, the package can be used only on a Linux machine, because of the use of makefile, which uses Conda because of the use in boost-python, and is installed from pip.
I tried to use setuptools.extensions.Extension but I am not really familiar with it, and I am not sure it is the right way because /cppCode has python files in it.
Therefore, I am almost sure that the problem is in setup.py.
python 3.8
Linux
anaconda3
g++
*UPDATE*
The SyntaxError happened due to a wrong relative import. I changed the import in wrapper.py to:
from . import _C as c
Now I do not get any error in the pip install.
I wrote test.py which uses wrapper.py and I get the following
ImportError:
Traceback (most recent call last):
File "/Users/user/test.py", line 1, in <module>
import myModule.cppCode.wrapper as wrap
File "/Users/user/anaconda3/lib/python3.8/site-
packages/myModule/cppCode/wrapper.py", line 1, in <module>
from . import _C as c
ImportError: cannot import name '_C' from 'myModule.cppCode' (/Users/user/anaconda3/lib/python3.8/site-packages/myModule/cppCode/wrapper.py__init__.py)
Although I am not getting an error in pip install, the .so file is still not being created after the pip install.