Flask multi-process worker pool in Python

Viewed 6485

I am pretty confused on the best way to do what I am trying to do.

I have a series of jobs that take a lot of time to process and so I would like to initialize a substantial pool of "workers" on app start up that will be able to handle a request when free.

Breakdown of the process:

  • Create 10 classifiers (workers) on app start up and keep them idle somewhere.
  • When app started a request comes through Flask POST command.
  • Request is passed to an available classifier.
  • Classifier returns result of job.

How would I go about doing this?

EDIT: It is worth noting that the classifiers take a substantial amount of time to start up and so will need to be available and already running when work it passed to them.

class View(views.MethodView):
    def get(self):
        return render_template('index.html')

    def post(self):
        app_form_elements = request.form

    #Assumption button clicked on browser interface...

            jobs = ["job one", "job two", "job three"]

            for job in jobs: 
        #Send each job to next available classifier pool.



        return self.get()



app.add_url_rule('/', view_func=View.as_view('main'), methods=['GET','POST'])

app.debug = True

if __name__ == '__main__':
    app.run(threaded=True)

EDIT:

The classifier setup is something like this:

class Classifier():
    """
    Class will take in a classifier and a test data set and print out the overall accuracy.
    """
    def __init__(self):

        self.load = self.toSomeStuff()

        print('classifier initialised.\n')


    def doSomeWork(self):
    #Initialised classifier objects called with work to do.



initialise_classifier = Classifier()

#the jobs 
initialise_classifier.doSomeWork()

So basically I need a pool of pre-initialized classifiers and then be able to call the "doSomeWork" function on them with each job that comes in via the post method.

2 Answers

You can't do this in Flask. But you can achieve the same result using, for example, uWSGI builtin functionalities or task queues like Celery.

Take a look at this answer for more details.

Related