Calling a Python C extension blocks all Django processes/users

user_2726773 0
Viewed 58

Problem

I have written a Python C extension (ftaCalculate) in order to improve the performance of a given function that was previously written in pure Python. I have been able to increase the execution speed by a factor of x10, so no problem on this site.

import ftaCalculate

cs_min = ftaCalculate.mcs(N, tree)

However, I am executing this function in a Django framework. The problem is that, until the function ftaCalculate.mcs does not finish, I cannot do anything on my website. When the function was in Python, I could press other buttons and access other URLs.

This is specially a problem when several users are working at the same time in the website, because the other users cannot do anything while this function is being executed.

Actually, you can see in the following image that one core is at 100% when running the function:

enter image description here

Question

Do you know any way I could call my Python C extension without "freezing" the Django framework?

Possible workaround

In the worst case, I could try calling this part of the code with Celery. However, I would prefer another solution, since I do not need Celery when running in pure Python.

multithreading c django python
1 Answers

As @Kemp suggested, the problem was on the Global Interpreter Lock (GIL). The solution has been to release the GIL when I am not using any Python object and reacquire it afterwards.

Following the details of Releasing the GIL from extension code section, I have put the corresponding C code between the following two lines:

Py_BEGIN_ALLOW_THREADS
... Code dealing only with C objects ...
Py_END_ALLOW_THREADS

This allows to call other processes within Django while the function is being executed.

user_27267730
Related