How can I use an external C library wrapped with Cython in a separate Python process via multiprocessing?

user_1062535 0
Viewed 217

I have wrapped a small C library in Cython, and can successfully call it from Python. Here's a simplified example representing how this is as of now:

# wrapper_module.pyx

cdef extern from "my_c_library.h":
    void function1()
    int function2(int param1, char *param2)

class MyWrapperClass():
    def __init__(self):
        pass
    def do_func1(self):
        function1()
    def do_func2(self, p1, p2):
        function2(p1, p2)

This all works well. My goal now is to create and use an instance of MyWrapperClass in a separate process, like this:

# my_script.py

import multiprocessing as mp
from wrapper_module import MyWrapperClass

class MyProcess(mp.Process):
    def __init__(self):
        super().__init__()

    def run(self):
        self.mwc = MyWrapperClass()
        self.mwc.do_func1()
        # ... do more

if __name__ == '__main__':
    proc = MyProcess()
    proc.start()

When I run my_script.py, I get the following error:

The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC() to debug.

I think I understand generally why it does not allow this cython module to be forked into a different process (due to resources that the underlying C library uses). But generally when I have encountered this type of problem in the past, either in pure Python or using ctypes and calling DLLs, I can solve it by placing all the critical code in the run method of MyProcess, causing it only to be initialized in the newly forked process.

In this Cython case, however, I don't know how to include the cdef extern code only in the forked process, to avoid this error. Any suggestions?

(I am running python 3.6.2 on macOS 10.12)

python-multiprocessing cython python-3.x c python
1 Answers

Codes in cdef extern are just delarations for external c functions(Let's only consider c), these delarations are put into the generated c file without any changes(almost), it has nothing to do with initializations.

I also made a rather simple test(py3.6 windows):

lib.c:

int add(int a, int b)
{
    return a + b ;
}

mylib.pyx:

cdef extern from "lib.c":
    int _add "add"(int a, int b)

cdef class WrapperClass:
    def add(self, a, b):
        return _add(a, b)

test_lib.py:

from mylib import WrapperClass
import multiprocessing as mp

class MyProcess(mp.Process):
    def __init__(self):
        super().__init__()

    def run(self):
        self.mwc = WrapperClass()
        print(self.mwc.add(1, 2))


if __name__ == "__main__":
    proc = MyProcess()
    proc.start()

run python test_lib.py without any problem. Maybe my c code is too simple, but I have no idea what your c functions realy do, myabe that's the real problem.

user_66100310
Related