Place pre-compiled extensions in root folder of non-pure Python Wheel package

user_1201456 0
Viewed 383

TL;DR How do you get distutils/setuptools to include a non-pure data file correctly?

I've got a project that does some non-trivial code generation using a custom toolchain, and then wraps that generated code with SWIG, and finally builds Python extensions. Cmake encapsulates all this excellently and in the end I have a single folder in the project's root that works exactly as any other Python package.

I'd like to have a simple setup.py so I can wrap this package into a wheel and ship it off to PyPI so that normal Python users don't have to deal with the build process. There are plenty of answers on SO for how to force setuptools to generate a non-pure wheel, and then you can bundle the extensions by using the package_data field or a MANIFEST.in file.

The problem is this method results in a malformed wheel, because the extensions get included under purelib instead of the root directory (where they belong in a Root-Is-Pure: False wheel). Some tools and distros rely on this seperation being correct.

Answers I'm not interested in: Custom extensions to run cmake from within setup.py (Don't want to add another layer of indirection for configuring the project, don't want to maintain it when build options change), modifying the generated wheel, and I'd prefer to avoid adding any more files to the project root than just setup.py

distutils setuptools python
1 Answers

This works. distutils and setuptools have to be some of the worst designed pieces of central Python infrastructure that exist.

from setuptools import setup, find_packages, Extension
from setuptools.command.build_ext import build_ext
import os
import pathlib
import shutil

suffix = '.pyd' if os.name == 'nt' else '.so'

class CustomDistribution(Distribution):
  def iter_distribution_names(self):
    for pkg in self.packages or ():
      yield pkg
    for module in self.py_modules or ():
      yield module

class CustomExtension(Extension):
  def __init__(self, path):
    self.path = path
    super().__init__(pathlib.PurePath(path).name, [])

class build_CustomExtensions(build_ext):
  def run(self):
    for ext in (x for x in self.extensions if isinstance(x, CustomExtension)):
      source = f"{ext.path}{suffix}"
      build_dir = pathlib.PurePath(self.get_ext_fullpath(ext.name)).parent
      os.makedirs(f"{build_dir}/{pathlib.PurePath(ext.path).parent}",
          exist_ok = True)
      shutil.copy(f"{source}", f"{build_dir}/{source}")

def find_extensions(directory):
  extensions = []
  for path, _, filenames in os.walk(directory):
    for filename in filenames:
      filename = pathlib.PurePath(filename)
      if pathlib.PurePath(filename).suffix == suffix:
        extensions.append(CustomExtension(os.path.join(path, filename.stem)))
  return extensions

setup(
  # Stuff
  ext_modules = find_extensions("PackageRoot"),
  cmdclass = {'build_ext': build_CustomExtensions}
  distclass = CustomDistribution
)

I'm copying extensions into the build directory, and that's it. We override the distribution to lie to the egg-info writers about having any extensions, and everything is gravy.

user_12014560
Related