I have a long running celery task running on remote host. After completion the result get() method is stuck waiting for result from the celery worker.
Rabbitmq is used as message broker and redis is used as backend.
I am calling apply_async() from flask, the job is submitted to celery worker on remote windows host.
Celery worker completes its task successfully printing
Task test[93a4d8a9-b6b9-4a54-8d65-365584ecba9d] succeeded in 9828.842999999993s
But the result get() method waits forever.
On celery broker side, I have mentioned visibility timeout as follows. Here app refers to celery instance.
app.conf.broker_transport_options = {"visibility_timeout": 14400, "max_retries": 3, "interval_start": 0, "interval_step": 0.2, "interval_max": 0.5}
On Celery client side inside get method I have mentioned timeout as follows:
doors_res = doors_res_obj.get(timeout=14400)
When timeout is not mentioned in get() of celery result, the application keeps on waiting for result in hang state.
With addition of timeout to get(), the connection reset error occurs. Error logs are as follows:-
Traceback (most recent call last):
File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\flask\app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\flask\app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\flask_restplus\api.py", line 325, in wrapper
resp = resource(*args, **kwargs)
File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\flask\views.py", line 89, in view
return self.dispatch_request(*args, **kwargs)
File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\flask_restplus\resource.py", line 44, in dispatch_request
resp = meth(*args, **kwargs)
File "C:\Users\venugopal.venkatesh\OneDrive - Veoneer\Documents\sprint35\Testing_with_Docker_env\flask\src\application\routes.py", line 89, in wrap
return f(*args, **kwargs)
File "C:\Users\venugopal.venkatesh\OneDrive - Veoneer\Documents\sprint35\Testing_with_Docker_env\flask\src\db\__init__.py", line
280, in wrap
return f(*args, **kwargs)
File "C:\Users\venugopal.venkatesh\OneDrive - Veoneer\Documents\sprint35\Testing_with_Docker_env\flask\src\doors\routes.py", line 127, in post
abort(503, ex)
File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\flask_restplus\errors.py", line 31, in abort
flask.abort(code)
File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\werkzeug\exceptions.py", line 772, in abort
return _aborter(status, *args, **kwargs)
File "C:\Users\venugopal.venkatesh\.virtualenvs\flask-2pC1ju1x\lib\site-packages\werkzeug\exceptions.py", line 753, in __call__
raise self.mapping[code](*args, **kwargs)
werkzeug.exceptions.ServiceUnavailable: 503 Service Unavailable: The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
2020-09-13 14:32:49,467 WARNING MainProcess - {'message': '[WinError 10054] An existing connection was forcibly closed by the remote host'}
**WARNING:debuglog:{'message': '[WinError 10054] An existing connection was forcibly closed by the remote host'}**
Any pointers on how to fix this issue?