Revoke multiple celery tasks

user_11719082 0
Viewed 161

I want to revoke multiple tasks from celery. From official docs, they suggested below approach. Is there any limit for this? Because i will get around 10k or 100k tasks to revoked.

>>> app.control.revoke([
...    '7993b0aa-1f0b-4780-9af0-c47c0858b3f2',
...    'f565793e-b041-4b2b-9ca4-dca22762a55d',
...    'd9d35e03-2997-42d0-a13e-64a66b88a618',
])
celery-task celery
2 Answers

There is not a limit. But please note the limitations around task revocation. In particular, this list of task IDs will be kept in memory or on disk, so there may be some amount of memory or disk overhead.

user_60431700

There is an undocumented limit in max 50000 flagged as revoked tasks stored in worker memory (as min for Celery 4.3 and 5.2)

Meaning that you can't handle more than 50000 flagged as revoked tasks per worker. Running Celery cluster will not help also.

While task discarded it removed from celery.worker.state.revoked and decrement flagged as revoked counter by one.

Related Celery source code celery.worker.state

Experimental approval:

  1. Run celery worker [optionally with --statedb path to create a file]:
$ celery worker --statedb=STATE_DB_FILEPATH
  1. Revoke 60000 tasks (by generated ids) in celery shell:
$ celery shell
>> from uuid import uuid4
>> REVOKE_TASK_NUMBER = 60000
>> celery.control.revoke([str(uuid4()) for i in range(REVOKE_TASK_NUMBER)])
  1. Validate that worker stores only 50000 revoked ids:
$ celery inspect revoked | wc -l
50003
  1. Shutdown the worker and validate that statedb stores only 50000 records (if created clean):
import zlib
import shelve

from kombu.serialization import pickle, pickle_protocol

filename=STATE_DB_FILEPATH

db = shelve.open(filename, protocol=pickle_protocol)
data = pickle.loads(zlib.decompress(db['zrevoked']))
print(len(data))

>> 50000
user_70068440
Related