Sharing huge classifier object among celery tasks

Viewed 1467

I have a pickle object ( classifier ) which is ~200MB. I tried to use static member variable to initialize the object and ran a celery task. Celery created ~8 new processes and each read the object into memory and froze my computer. Is there any shared object solution or any solution to this problem.

I'm trying to classify articles with this object and save it in db as a background task.

EDIT. Here is the code for my classifier Task

from model import Article
import cPickle as pkl
from classifierclasses import Remover, Classifier, ItemSelector, PreProcessor
from celery import Celery
app = Celery('tasks', backend='redis://localhost/')

class ClassiferTask(app.Task):
    classifier = pkl.load(open('clf.p', 'rb')) #loading classifier  object initially (huge file)
    def classify(self, article_id, text):
        self.prediction = ClassiferTask.classifier.predict([text])[0]
        Article.update(prediction=self.prediction, is_analyzed=1).where(Article.id == article_id).execute()

    def run(self, id, *args, **kwargs):
        self.classify(id, args[0])

when I submit this job to celery, it created new processes and each process trying to read into memory and occupying large memory ~200MB * 8

how can I solve this problem? Is there a better solution to this problem. Is there a celery task that can share a single (read only) object.

1 Answers
Related