`threading.local` unexpected behavior with Python embedding

Viewed 59

I am embedding Python using C embedding API. The main thread does

Py_Initialize();
PyEval_InitThreads();

Then I have more threads created by native code, whose lifetime I do not control. They need to call Python too. So far they seemed to be working fine with

void* gil = PyGILState_Ensure();

calls to Python go here

PyGILState_Release(gil);

Problem is given this simple setup I faced issues with Python code, that uses threading.local. Imagine secondary thread S, that periodically executes increase_counter:

// initialized once at the beginning of program to
// an instance of threading.local()
PyObject* threadLocal;

...

void increase_counter()
{

void* gil = PyGILState_Ensure();

// this really does C API calls, but for simplicity I'll write Python
if hasattr(threadLocal, "counter"):
  threadLocal.counter += 1
else:
  threadLocal.counter = 1
// end of Python

PyGILState_Release(gil);

}

Well, the problem is that in thread S multiple calls to increase_counter don't actually increase anything - hasattr always returns False, and the value of counter for this thread is discarded as soon as PyGILState_Release is called.

It only works correctly in S if the whole body of S is wrapped into:

void* gilForS = PyGILState_Ensure();

void* sPythonThreadState = PyEval_SaveThread();

// rest of the S body, that sometimes calls increase_counter

PyEval_RestoreThread(sPythonThreadState);

PyGILState_Release(gilForS);

Which I can do for the purpose of this question, but in the actual product the lifetime of S is not controlled by me (it is a thread pool thread), only increase_counter, so I can't make it run PyEval_SaveThread in the beginning, and I can't ensure PyEval_RestoreThread will be called in the end.

What is the proper way to initialize threads like S so that threading.local would correctly work there?

Full example, that reproduces the issue as requested. Prints "set counter" twice instead of "set counter" + "Found counter!". It works as expected if I uncomment code in async_thread which I can't do in real product:

#include <Python.h>
#include <pthread.h>

PyObject *threadLocal;
void *async_thread(void *arg);
void increase_counter();

int main(int argc, char *argv[])
{
    Py_Initialize();
    PyEval_InitThreads();

    PyObject *threading = PyImport_ImportModule("threading");
    PyObject *tlocal = PyObject_GetAttrString(threading, "local");
    threadLocal = PyObject_Call(tlocal, PyTuple_New(0), NULL);

    pthread_t async;
    int err = pthread_create(&async, NULL, async_thread, NULL);
    if (err)
    {
        printf("unable to create thread\n");
        exit(-1);
    }

    PyThreadState* ts = PyEval_SaveThread();

    pthread_join(async, NULL);

    PyEval_RestoreThread(ts);

    Py_Finalize();

    pthread_exit(NULL);
}

void *async_thread(void *arg)
{
    //PyGILState_STATE gil = PyGILState_Ensure();
    for (int i = 0; i < 2; i++)
    {
        increase_counter();
    }
    //PyGILState_Release(gil);
    pthread_exit(NULL);
    return NULL;
}

void increase_counter()
{
    PyGILState_STATE gil = PyGILState_Ensure();

    if (PyObject_HasAttrString(threadLocal, "counter"))
    {
        printf("Found counter!\n");
    }
    else
    {
        PyObject *val = PyLong_FromLong(1);
        PyObject_SetAttrString(threadLocal, "counter", val);
        printf("set counter\n");
    }

    PyGILState_Release(gil);
}
0 Answers
Related