I am running inference using a Huggingface transformer model for question answering.
I have exposed this as an API using flask and gunicorn (with multiple workers).
I have noticed that when a single request is issued, more than 1 CPU cores are being utilized.
Hence, when more than one request is issued concurrently, the response for each is taking much longer. For example, 1 request = 1.2 seconds whereas, 2 requests = getting both responses after 2 seconds.
It would be ideal if both requests took 1.2 seconds to be served.
This issue is occurring although I have more than one workers in gunicorn.
In this article, they address the issue by thread tuning in PyTorch, they offer a solution by adding this line to their code:
torch.set_num_threads(1)
It seems that this is supposed to ensure that each worker uses exactly one cpu core and make sure they are not competing for the same resources.
I have incorporated this into my code. However, the problem still seems to persist.
Hence, I suspect that this is due to some configuration in gunicorn.
The Flask app is defined in the file QA_transformer.py and this is what I use to start gunicorn:
gunicorn --bind 0.0.0.0:5000 QA_transformer:app --workers 3
Is there something that should be added or changed?
How can I make sure that each worker uses only one CPU core?
Or does the torch.set_num_threads(1) not work as expected for 'pipeline' in the transformers library from Huggingface?
PS. I am using a machine with 4 CPU cores.