Facing memory error on my Django celery worker instance

user_1900645 0
Viewed 407

I am using django celery with redis (broker). I have observed following error on one of my worker instance.

[2020-12-27 02:26:15,920: INFO/MainProcess] missed heartbeat from worker@ip-xxx-xx-xx-
xxx.ec2.internal
[2020-12-27 02:26:40,937: INFO/MainProcess] missed heartbeat from worker@ip-xxx-xx-xx-xxx.ec2.internal
[2020-12-27 02:27:00,943: INFO/MainProcess] missed heartbeat from worker@ip-xxx-xx-xx-xxx.ec2.internal
[2020-12-27 02:27:15,955: INFO/MainProcess] missed heartbeat from worker@ip-xxx-xx-xx-xxx.ec2.internal
[2020-12-27 02:27:45,971: INFO/MainProcess] missed heartbeat from worker@ip-xxx-xx-xx-xxx.ec2.internal
[2020-12-27 02:28:02,118: INFO/MainProcess] missed heartbeat from worker@ip-xxx-xx-xx-xxx.ec2.internal
[2020-12-27 02:28:36,496: CRITICAL/MainProcess] Unrecoverable error: MemoryError()
Traceback (most recent call last):
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 596, in start
    c.loop(*c.loop_args())
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 364, in create_loop
    cb(*cbargs)
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/kombu/transport/redis.py", line 1074, in on_readable
    self.cycle.on_readable(fileno)
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/kombu/transport/redis.py", line 359, in on_readable
    chan.handlers[type]()
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/kombu/transport/redis.py", line 694, in _receive
    ret.append(self._receive_one(c))
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/kombu/transport/redis.py", line 700, in _receive_one
    response = c.parse_response()
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/redis/client.py", line 3036, in parse_response
    return self._execute(connection, connection.read_response)
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/redis/client.py", line 3013, in _execute
    return command(*args)
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/redis/connection.py", line 637, in read_response
    response = self._parser.read_response()
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/redis/connection.py", line 330, in read_response
    response = [self.read_response() for i in xrange(length)]
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/redis/connection.py", line 330, in <listcomp>
    response = [self.read_response() for i in xrange(length)]
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/redis/connection.py", line 324, in read_response
    response = self._buffer.read(length)
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/redis/connection.py", line 205, in read
    self._read_from_socket(length - self.length)
  File "/home/ec2-user/.virtualenvs/xxxxx/lib/python3.7/site-packages/redis/connection.py", line 186, in _read_from_socket
    buf.write(data)
MemoryError
[2020-12-27 06:44:31,570: INFO/MainProcess] Connected to redis://xxxxxxxxxx.cache.amazonaws.com:6379//
[2020-12-27 06:44:31,585: INFO/MainProcess] mingle: searching for neighbors
[2020-12-27 06:44:32,611: INFO/MainProcess] mingle: sync with 1 nodes

I just want to confirm, either this memory error is due memory leakage in my code somewhere, some worker specific issue, or due to some other reason. I really appreciate any help/suggestion to figure out root cause.

Note: Instance type of my worker (on aws) is t2.small

celerybeat celery redis django
1 Answers

Make sense (small instance) but I more worried about health-checks failures (missing heartbeats).

here are some ideas:

  • Try to profile your celery task to understand how much memory it consumes. Is it more than the 2GB of this instance type?
  • What's the concurrency level you defined for your worker? did you try to decrease this number? if c==2 and every task consumes 2GB (for example) that's can explain your issue.
  • Use CloudWatch metrics (in AWS console) to see the CPU & Mem Utilization to see whether you find a correlation between your error time and some spikes in the graphs.
  • If it is reproducible you can try htop while having this error - to ensure that's a resource limitation (mem/CPU).
  • Collect those metrics yourself - it can always help you for such cases.

Good luck!

user_10112530
Related