I can't seem to figure this out. I've got a model trained with scikit-learn, saved to a .pkl file, and I want to make an API that would make predictions based on it.
I already have the code that makes predictions and it runs fine from console/unit-tests. To speed up predictions I'm splitting the data (thousands of image patches) and spreading the load using joblib/multiprocessing.
I'm setting JOBLIB_START_METHOD=forkserver since scikit-learn hangs if used from within a multiprocessing process.
I've got an API done with flask which uses this code, and when run with flask's dev server it works just fine. Now I'm trying to host the flask app within gunicorn and it's not working at all.
If I use the default workers, then it just hangs with no errors when trying to predict, much like if I hadn't set the 'forkserver' multiprocessing. I'm running gunicorn like this:
JOBLIB_START_METHOD=forkserver gunicorn -w 2 -b 0.0.0.0:$PORT --timeout 3600 web.app:app
I also tried using the gevent backend. This actually does work but it's very slow, and it prints this:
Multiprocessing backed parallel loops cannot be nested below threads, setting n_jobs=1
So, any ideas on getting this to work in a way that there's multiple web workers running (I don't think that's the case with flask's dev server) and with a request being able to leverage joblib/multiprocessing? thanks