How can I run cython'd shared libraries on Google Cloud Functions?

user_992509 0
Viewed 493

As the title says, I was wondering if Google Cloud Functions (where I currently have some pure python code) support cython'd modules?

I guess, more specifically, I'm asking about how I would use said modules? It's a private project, I'm using cython via setup.py and cythonize(files) which creates a bunch of shared object modules (example.cpython-38-darwin.so, example1.cpython-38-darwin.so, example2.cpython-38-darwin.so).

Those are all for Mac, so won't work on Firebase.

Is there any way to get Cloud Functions to run the setup.py and compile some files? Or, better yet, is there some way to pre-compile those files for the appropriate OS and just deploy the shared libs?

I know a variety of libraries I'm installing via pip on Cloud Functions use Cython under the hood, but I don't really know the process of creating a wheel or other pip dependency...

google-cloud-functions cython google-cloud-platform python
2 Answers

You need to specify cython as a build-time dependency for your private project by adding a pyproject.toml file like:

[build-system]
requires = ["cython"]

Then when installing your package with the modern version of pip in the Cloud Functions runtime, cython will be installed into the build environment before your setup.py script is run.

user_3280360

I appear to have been able to (eventually) solve this... I might have several unnecessary steps, but I think they improve my overall build system (again, with the intention of being able to use cython'd shared libraries on Firebase).

From Docker (or in my case, a Linux VM), in my private repo, I cythonize the important code, and turn everything into a wheel. From here, I run auditwheel show over the wheel, to check if it adheres to the manylinux1 tag (or whichever manylinux I want). In this case, it did adhere to manylinux1 off the bat, so there was no need to repair the wheel or do any shenanigans this time.

... .py # Other irrelevant .py files
magic.py # Source code that needs to be cython'd
setup.py

Simplified setup.py:

from setuptools import setup, find_packages
from Cython.Build import cythonize

setup(
    name='magiclib',
    version='0.1.0',
    packages=find_packages(),
    ext_modules=cythonize(
        "magic.py",
        compiler_directives={'language_level': 3}
    )
)

Running python setup.py bdist_wheel creates a wheel named dist/magiclib-0.1.0-cp37-cp37m-linux_x86_64.whl

From here, I run auditwheel show dist/magiclib-0.1.0-cp37-cp37m-linux_x86_64.whl which shows me that the code already adheres to the manylinux1 tag, but I nonetheless run auditwheel repair dist/magiclib-0.1.0-cp37-cp37m-linux_x86_64.whl which creates wheelhouse/magiclib-0.1.0-cp37-cp37m-manylinux1_x86_64.whl.

At this point, I bring this wheel into my GCF project, and use: pip install -t magiclib magiclib-0.1.0-cp37-cp37m-manylinux1_x86_64.whl

which basically unzips the wheel into a sub-directory that I can vendor and deploy to Google Cloud and call from my Functions.

Works fine on some of my simple code, and I'll be experimenting with some more involved code.

user_9925090
Related