Celery async_result.get() hangs waiting for result even after celery worker has completed the task

Viewed 1015

I have a long running celery task running on remote host. After completion the result get() method is stuck waiting for result from the celery worker.

Rabbitmq is used as message broker and redis is used as backend.

I am calling apply_async() from flask, the job is submitted to celery worker on remote windows host. Celery worker completes its task successfully printing Task test[93a4d8a9-b6b9-4a54-8d65-365584ecba9d] succeeded in 9828.842999999993s

But the result get() method waits forever.

On celery broker side, I have mentioned visibility timeout as follows. Here app refers to celery instance.

app.conf.broker_transport_options = {"visibility_timeout": 14400, "max_retries": 3, "interval_start": 0, "interval_step": 0.2, "interval_max": 0.5}

On Celery client side inside get method I have mentioned timeout as follows:

doors_res = doors_res_obj.get(timeout=14400)

When timeout is not mentioned in get() of celery result, the application keeps on waiting for result in hang state.

With addition of timeout to get(), the connection reset error occurs. Error logs are as follows:-


Traceback (most recent call last):
  File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\flask\app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\flask\app.py", line 1936, in dispatch_request   
    return self.view_functions[rule.endpoint](**req.view_args)
  File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\flask_restplus\api.py", line 325, in wrapper    
    resp = resource(*args, **kwargs)
  File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\flask\views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\flask_restplus\resource.py", line 44, in dispatch_request
    resp = meth(*args, **kwargs)
  File "C:\Users\venugopal.venkatesh\OneDrive - Veoneer\Documents\sprint35\Testing_with_Docker_env\flask\src\application\routes.py", line 89, in wrap
    return f(*args, **kwargs)
  File "C:\Users\venugopal.venkatesh\OneDrive - Veoneer\Documents\sprint35\Testing_with_Docker_env\flask\src\db\__init__.py", line 
280, in wrap
    return f(*args, **kwargs)
  File "C:\Users\venugopal.venkatesh\OneDrive - Veoneer\Documents\sprint35\Testing_with_Docker_env\flask\src\doors\routes.py", line 127, in post
    abort(503, ex)
  File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\flask_restplus\errors.py", line 31, in abort    
    flask.abort(code)
  File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\werkzeug\exceptions.py", line 772, in abort     
    return _aborter(status, *args, **kwargs)
  File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\werkzeug\exceptions.py", line 753, in __call__  
    raise self.mapping[code](*args, **kwargs)
werkzeug.exceptions.ServiceUnavailable: 503 Service Unavailable: The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
2020-09-13 14:32:49,467 WARNING MainProcess - {'message': '[WinError 10054] An existing connection was forcibly closed by the remote host'}
**WARNING:debuglog:{'message': '[WinError 10054] An existing connection was forcibly closed by the remote host'}**

Any pointers on how to fix this issue?

0 Answers
Related