Celery get state of two tasks at the same time not working

Viewed 189

I have an asynchronous API with flask, celery and RabbitMQ configured as a backend and broker. I have three endpoints, /task1, /task2 and /progress/<task_id>.

In /task1 and /task2 I have something like this:

    task = task1.delay()
    return task.id

And /progress/<task_id>:

@app.route('/progress/<task_id>', methods=['GET'])
def progress(task_id):
  def progress_stream():
    p = ""
    while p != "SUCCESS":
      res = cel_app.AsyncResult(task_id, app=cel_app)
      p = res.status
      yield p

  return Response(progress_stream(), mimetype='text/event-stream')

And in the client I have the following script:

var eventSource = new EventSource("http://localhost:5000/progress/[task_id]")
  eventSource.onmessage = function(e) {
  container.innerHTML = e.data;
  console.log(e.data)
  if(e.data == "SUCCESS"){
    eventSource.close();
  }
};

And I run celery with the following command:

 celery -A celery_app worker --loglevel=info --concurrency=10

This works fine when I make a single POST request to any of the task and then wait for its status. The problem comes when I make two or more request to /task1 or /task2 and then try to get the result of both tasks at the same time. In this case, it works with the first request, but the second one never updates the status or the result, even when celery is actually executing it.

I have read the documentation and I have found the following:

The RPC result backend (rpc://) is special as it doesn’t actually store the states, but rather sends them as messages. This is an important difference as it means that a result can only be retrieved once, and only by the client that initiated the task. Two different processes can’t wait for the same result.

However, in this case the two clients are waiting for different tasks, with different ids, so I am not sure why this is happening.

I also tried adding this to the celery configuration:

result_persistent=True

But it doesn't seem to make a difference

0 Answers
Related