Celery task with metadata - where to store results?

Viewed 1627

I have a long-running celery tasks which spits out some files. The result of the task is the folder location where the files are stored and the run time.

e.g:

@app.task()
def my_task():
   start = time.time()
   folder = my_heavy_function()
   end = time.time()
   return json.dumps({ 'folder': folder, 'time': end-start })

Before the async call to the task, I store metadata in mongodb - the user who is starting the task, description and so forth.

e.g.

def endpoint(req, resp, user, description):

   doc = MyMongoDbDocument(
             user=user
             description=description
         )
   task = my_task.delay()
   doc.task_id = task.id
   doc.save()
   resp.status = HTTP_OK
   resp.body = doc.to_json()

For persisting the results in this situation, I can think of two possible approaches:

  1. Use a celery backend to store the results (e.g. redis). In order to link the result to the metadata, store the task ID in the metadata.

  2. Have the task update the metadata with the result data.

What is the best practice here? I can see advantages and disadvantages in both situations:

  1. It means that the client will have to make two calls to retrieve all information related to the task. 1 to get the metadata and task id and another to some endpoint which will get the AsyncResult for the task id and return the result data. It also means less data locality. Related data is in different places. On the other hand, it means no duplication of data and code locality - i.e I am not accessing the (metadata) database in various locations.

  2. 1 call to get all data. However, it means passing the mongodb document id to the task, opening another connection to the database and updating it. There is maybe potential for write conflicts here. Also, even assuming all the db access code is neatly abstracted in some library, I think it is not quite as DRY as 1.

0 Answers
Related