I am not able to figure this out. I have flask app with celery and using Redis as a broker on Ubuntu 16.04. It was working fine for the last many days and today all of a sudden it is giving the following exception. Works fine locally on my OSX. Fails on the ubuntu virtual server.
[2017-07-26 20:12:25,512: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 129, in _read_from_socket
raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 2165, in _execute
return command(*args)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 577, in read_response
response = self._parser.read_response()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 238, in read_response
response = self._buffer.readline()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 168, in readline
self._read_from_socket()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 143, in _read_from_socket
(e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 439, in connect
sock = self._connect()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 494, in _connect
raise err
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 482, in _connect
sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/celery/worker/consumer/consumer.py", line 318, in start
blueprint.start(self)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/celery/worker/consumer/consumer.py", line 594, in start
c.loop(*c.loop_args())
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/celery/worker/loops.py", line 88, in asynloop
next(loop)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/kombu/async/hub.py", line 345, in create_loop
cb(*cbargs)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/kombu/transport/redis.py", line 1039, in on_readable
self.cycle.on_readable(fileno)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/kombu/transport/redis.py", line 337, in on_readable
chan.handlers[type]()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/kombu/transport/redis.py", line 667, in _receive
ret.append(self._receive_one(c))
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/kombu/transport/redis.py", line 678, in _receive_one
response = c.parse_response()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 2183, in parse_response
return self._execute(connection, connection.read_response)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 2172, in _execute
connection.connect()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 442, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.
[2017-07-26 20:12:25,804: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379//: Error 111 connecting to localhost:6379. Connection refused..
Trying again in 2.00 seconds...
[2017-07-26 20:12:26,725: ERROR/PoolWorker-3] Something went wrong in async processes.
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/tasks.py", line 66, in send_event_notification_async
sleep(nextEventDelayInSeconds)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/billiard/common.py", line 125, in _shutdown_cleanup
sys.exit(-(256 - signum))
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/billiard/pool.py", line 281, in exit
return _exit()
SystemExit
[2017-07-26 20:12:26,726: ERROR/PoolWorker-4] Something went wrong in async processes.
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/tasks.py", line 66, in send_event_notification_async
sleep(nextEventDelayInSeconds)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/billiard/common.py", line 125, in _shutdown_cleanup
sys.exit(-(256 - signum))
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/billiard/pool.py", line 281, in exit
return _exit()
SystemExit
[2017-07-26 20:12:26,734: ERROR/PoolWorker-8] Something went wrong in async processes.
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/tasks.py", line 66, in send_event_notification_async
sleep(nextEventDelayInSeconds)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/billiard/common.py", line 125, in _shutdown_cleanup
sys.exit(-(256 - signum))
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/billiard/pool.py", line 281, in exit
return _exit()
SystemExit
[2017-07-26 20:12:27,729: ERROR/MainProcess] Process 'PoolWorker-6' pid:2057 exited with 'signal 15 (SIGTERM)'
[2017-07-26 20:12:27,740: ERROR/MainProcess] Process 'PoolWorker-5' pid:2055 exited with 'exitcode 15'
[2017-07-26 20:12:27,750: ERROR/MainProcess] Process 'PoolWorker-2' pid:2052 exited with 'exitcode 15'
[2017-07-26 20:12:27,761: ERROR/MainProcess] Process 'PoolWorker-1' pid:2051 exited with 'exitcode 15'
[2017-07-26 20:12:27,812: ERROR/MainProcess] Process 'PoolWorker-7' pid:2059 exited with 'exitcode 15'
[2017-07-26 20:13:27,799: ERROR/PoolWorker-3] Something went wrong in async processes.
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 129, in _read_from_socket
raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 573, in execute_command
return self.parse_response(connection, command_name, **options)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
response = connection.read_response()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 577, in read_response
response = self._parser.read_response()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 238, in read_response
response = self._buffer.readline()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 168, in readline
self._read_from_socket()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 143, in _read_from_socket
(e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 439, in connect
sock = self._connect()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 494, in _connect
raise err
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 482, in _connect
sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/tasks.py", line 45, in send_event_notification_async
r.set(priority, True)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 1072, in set
return self.execute_command('SET', *pieces)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 578, in execute_command
connection.send_command(*args)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 563, in send_command
self.send_packed_command(self.pack_command(*args))
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 538, in send_packed_command
self.connect()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 442, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.
[2017-07-26 20:13:27,800: ERROR/PoolWorker-4] Something went wrong in async processes.
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 129, in _read_from_socket
raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 573, in execute_command
return self.parse_response(connection, command_name, **options)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
response = connection.read_response()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 577, in read_response
response = self._parser.read_response()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 238, in read_response
response = self._buffer.readline()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 168, in readline
self._read_from_socket()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 143, in _read_from_socket
(e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 439, in connect
sock = self._connect()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 494, in _connect
raise err
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 482, in _connect
sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/tasks.py", line 45, in send_event_notification_async
r.set(priority, True)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 1072, in set
return self.execute_command('SET', *pieces)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 578, in execute_command
connection.send_command(*args)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 563, in send_command
self.send_packed_command(self.pack_command(*args))
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 538, in send_packed_command
self.connect()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 442, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.
[2017-07-26 20:13:27,802: ERROR/PoolWorker-8] Something went wrong in async processes.
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 129, in _read_from_socket
raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 573, in execute_command
return self.parse_response(connection, command_name, **options)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
response = connection.read_response()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 577, in read_response
response = self._parser.read_response()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 238, in read_response
response = self._buffer.readline()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 168, in readline
self._read_from_socket()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 143, in _read_from_socket
(e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 439, in connect
sock = self._connect()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 494, in _connect
raise err
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 482, in _connect
sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dsangvikar/apps/msbot/tasks.py", line 45, in send_event_notification_async
r.set(priority, True)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 1072, in set
return self.execute_command('SET', *pieces)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/client.py", line 578, in execute_command
connection.send_command(*args)
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 563, in send_command
self.send_packed_command(self.pack_command(*args))
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 538, in send_packed_command
self.connect()
File "/home/dsangvikar/apps/msbot/msbotenv/lib/python3.5/site-packages/redis/connection.py", line 442, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.
I guess it retries after 2 seconds and again fails and the same exceptions. What is more confusing is redis-server is up and running properly. redis-cli ping works. Further more, I opened up redis-cli and tried to get the values which the celery worker is supposed to add. Those are also gettable. Btw, I flush all the values on server restart. So when I restarted the server, it updated the values accordingly. How do I figure out what's wrong?
I didn't make any changes on the redis-server configuration files. All ways working well till yesterday.
edit: What is suprising is that I get this error 'ConnectionRefusedError: [Errno 111] Connection refused' after some of the statements have been executed. So basically it connects to the redis db, puts in the values and then fails. What?