How to setup celery as daemon

user_11971890 0
Viewed 1515

I am faced with necessity to setup celery as daemon for my django project on Ubuntu 16.04 server while doing it I met several misunderstandings which I will describe in my question. I know that by the rules of Stack asked should should ask only one clear question but I will ask several in one question because first question come from second etc.

For tune celery as daemon I decide to use SystemD. In documentation Demonization celery provide a guide but it isn't so clear as I want maybe it's because I am beginner. My first question is: should I setup separately celery and celerybeat? Here is documentation provided example of configuration 

[Unit] 
Description=Celery Service    
After=network.target 

[Service] 
Type=forking 
User=celery 
Group=celery 
EnvironmentFile=/etc/conf.d/celery 
WorkingDirectory=/opt/celery 
ExecStart=/bin/sh -c '${CELERY_BIN}    multi start ${CELERYD_NODES} \ -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \ --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}' 
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \ --pidfile=${CELERYD_PID_FILE}'  
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \ -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \ --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}' 

[Install] WantedBy=multi-user.target

All my misunderstandings relate to this piece of code and next question is:

What user and group I should specify in Service part of configuration? in documentation example above celery specified for user and group but when I checked users and groups on my Ubuntu I haven't celery user and group should I create this kind of user and group if answer is yes what rights I should provide for celery user and group? Or I should specify in user and group settings my Ubuntu user under which I am working on server?

Next question is related to all variables which used to specify different parts of settings for example in above configuration example in ExecStart settings have multiple variables 

${CELERY_BIN}   
${CELERYD_NODES} 
${CELERY_APP} 
${CELERYD_PID_FILE} 
${CELERYD_LOG_FILE} 
${CELERYD_LOG_LEVEL} 
${CELERYD_OPTS}'

Where I should specify this variables I think that it should be in django settings.py file but documentation above provide example which confused me maybe you can think why I can't to try configure this variables in settings.py rather than asking it here but I can't to test it because I don't have understanding about user and group question

Next question is about this setting

EnvironmentFile=/etc/conf.d/celery

What I should specify here because I have not this file on my system? Should I create it? It's seems that is default settings for celery but I use django and all related settings to celery I specified in my django project settings.py file

Can anyone guide me? Thank in advance

celery django
2 Answers

I too struggled to understand and implement the instructions given in celeryproject.org's daemonisation guide for systemd. Finally I just used general daemonisation instructions and it worked. I am using Flask, but I suppose this should work for you also :

This is my celery.service file. I have flask and celery running in a virtual environment (venv) :

[Unit]
Description=Celery Service
After=network.target

[Service]
User=your_username
WorkingDirectory=/home/your_username/your_projdir
Environment="PATH=/home/your_username/your_projdir/venv/bin"
ExecStart=/home/your_username/your_projdir/venv/bin/celery worker -A celery_worker.celery --loglevel=info

[Install]
WantedBy=default.target

You could add more commands here, and choose to include EnvironmentFile, but this simple setup worked well for me.

user_60390950

Yes, create new user (celery is a good name). No need for any special attributes. Regular user should be fine. You define the necessary environment variables inside the /etc/conf.d/celery file.

Let's say you have created celery user in /home/celery... Log-in as that user, and create Python 3 virtual environment: python3 -m venv ~/venv. After that your /etc/conf.d/celery should have something like:

CELERY_BIN=/home/celery/venv/bin/celery   
CELERY_APP=myproject.myapp               # change this to however you named it
CELERY_OPTS=-Ofair -c12                  # any other options here

You need to define here all the vars you used in your systemd service file.

Also, there is no need for /bin/sh -c in Exec{Start/Stop/Reload} - ${CELERY_BIN} multi ... will work as ${CELERY_BIN} should point to the Celery script in your virtual environment, which is executable.

user_8764970
Related