How to use periodic tasks with django-celery-beat in two Django projects sharing the same database

Florent 0
Viewed 56

I have two Django projects sharing the same PostgreSQL database. Celery and django-celery-beat application are created and installed in both projects.

The problem is that periodic tasks from one project are detected by the Celery application of the other project as unregistered and REJECTED. I believe the django-celery-beat of both projects share the same tables of the database.

I came to this conclusion because both projects are containerized, they have their own Docker network and redis server listening on 2 different ports number. Finally, both projects have a dedicated default queue name.

What is the best way to address the issue ? I'm relatively new with Django but I think each project should have a dedicated django-celery-beat application. Can I do that with AppConfig and then create new tables with migrate ?

The naming could be something like :

settings.py (project A)

INSTALLED_APPS = [
    ...
    'django_celery_beat_project_a',
    ...
]

settings.py (project B)

INSTALLED_APPS = [
    ...
    'django_celery_beat_project_b',
    ...
]

Output

project-a-celery_worker-1  | 2022-09-09T11:35:15.974463294Z Received unregistered task of type 'trading.tasks.bulk_trade'.
project-a-celery_worker-1  | 2022-09-09T11:35:15.974621504Z The message has been ignored and discarded.
project-a-celery_worker-1  | 2022-09-09T11:35:15.974655749Z 
project-a-celery_worker-1  | 2022-09-09T11:35:15.974677303Z Did you remember to import the module containing this task?
project-a-celery_worker-1  | 2022-09-09T11:35:15.974699464Z Or maybe you're using relative imports?
project-a-celery_worker-1  | 2022-09-09T11:35:15.974721029Z 
project-a-celery_worker-1  | 2022-09-09T11:35:15.974741804Z Please see
project-a-celery_worker-1  | 2022-09-09T11:35:15.974782954Z http://docs.celeryq.org/en/latest/internals/protocol.html
project-a-celery_worker-1  | 2022-09-09T11:35:15.974815238Z for more information.
project-a-celery_worker-1  | 2022-09-09T11:35:15.974839132Z 
project-a-celery_worker-1  | 2022-09-09T11:35:15.974897495Z The full contents of the message body was:
project-a-celery_worker-1  | 2022-09-09T11:35:15.974938229Z b'[[], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]' (77b)
project-a-celery_worker-1  | 2022-09-09T11:35:15.974982482Z 
project-a-celery_worker-1  | 2022-09-09T11:35:15.975032349Z Thw full contents of the message headers:
project-a-celery_worker-1  | 2022-09-09T11:35:15.975090058Z {'lang': 'py', 'task': 'trading.tasks.bulk_trade', 'id': 'd86620c2-04ab-4b59-bb2e-77685389f8d5', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd86620c2-04ab-4b59-bb2e-77685389f8d5', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': '{}', 'origin': 'gen9@03298c28290e', 'ignore_result': False}
project-a-celery_worker-1  | 2022-09-09T11:35:15.975125751Z 
project-a-celery_worker-1  | 2022-09-09T11:35:15.975194550Z The delivery info for this task is:
project-a-celery_worker-1  | 2022-09-09T11:35:15.975257110Z {'exchange': '', 'routing_key': 'project-a_queue_1'}
project-a-celery_worker-1  | 2022-09-09T11:35:15.981241271Z  exc_info=(<class 'KeyError'>, KeyError('trading.tasks.bulk_trade'), <traceback object at 0x7fe49ca1b8c0>)
project-a-celery_worker-1  | 2022-09-09T11:35:15.996976839Z [error    ] task_not_found                              [django_structlog.celery.receivers] message=<Message object at 0x7fe49cb6e680 with details {'state': 'REJECTED', 'content_type': 'application/json', 'delivery_tag': '43925569-3d7a-4272-9629-2e08581f6579', 'body_length': 77, 'properties': {'correlation_id': 'd86620c2-04ab-4b59-bb2e-77685389f8d5'}, 'delivery_info': {'exchange': '', 'routing_key': 'project-a_queue_1'}}>
celerybeat celery django python
1 Answers

I finally came to a solution and the problem is fixed. This is how to proceed.

Delete all periodic tasks.

Rename django_celery_beat in all the Python packages, I did it from the Dockerfile :

WORKDIR /usr/local/lib/python3.10/site-packages

RUN find . -type f -print0 | xargs -0 sed -i 's/django_celery_beat/django_celery_beat_project_a/g' \
    && sed -i "126 i \ \ \ \ name = 'django_celery_beat_trader.schedulers' if name == 'django_celery_beat.schedulers' else name" /usr/local/lib/python3.10/importlib/__init__.py \
    && cp -r django_celery_beat django_celery_beat_project_a

Rename the application in settings.py:

INSTALLED_APPS = [
    ...
    'django_celery_beat_project_a',
    ...
]

Write the new tables in the database:

python manage.py makemigrations
python manage.py migrate
Florent0
Related