Why does adding an __init__.py change Cython build_ext --inplace behavior?

user_388027 0
Viewed 168

This is a toy version of a problem I have. I'm trying to use setuptools to compile some Cython code in place, as part of a larger project.

My topdir is test. It contains:

hello_world.pyx

def say_hi():
     print('meh')

and setup.py

from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules=cythonize("hello.pyx"),
)

If I build these modules in place with python3 setup.py build_ext --inplace, everything works as expected.

But if add an empty __init__.py to my topdir, I get the following error:

copying build/lib.linux-x86_64-3.8/Code/test/hello.cpython-38-x86_64-linux-gnu.so -> Code/test
error: could not create 'Code/test/hello.cpython-38-x86_64-linux-gnu.so': No such file or directory

Why is this happening? And is there a standard way of fixing it? I would like for these compiled Cython modules to live alongside some other C/Python code which is still in development.

cython setuptools
1 Answers

what is happening?

The complete description of what is happening can be found in this GitHub issue: https://github.com/pypa/setuptools/issues/3343#issuecomment-1143527672

TL;DR: When you add a __init__.py file in your top folder, it interferes with the way Cython computes the extension's name. This results in an incorrect name (incompatible with your project layout due to a missing folder) and causes setuptools to crash.

And is there a standard way of fixing it?

If you really want to keep the __init__.py file under your top-level project folder[^1], you can explicitly use Extension objects with the proper/correct name instead of relying on the automatic naming algorithm used by Cython:

from setuptools import Extension, setup
from Cython.Build import cythonize

extensions = [Extension("hello", ["hello.pyx"])]

setup(
    ext_modules=cythonize(extensions),
)

[^1] Please note that this is not common practice in the Python packaging ecosystem.

user_41671710
Related