How to use Airflow scheduler with systemd?

user_10163853 0
Viewed 6489

The docs specify instructions for the integration

What I want is that every time the scheduler stop working it will be restarted by it's own. Usually I start it manually with airflow scheduler -D but sometimes it stops when I'm not available.

Reading the docs I'm not sure about the configs.

The GitHub contains the following files:

airflow
airflow-scheduler.service
airflow.conf

I'm running Ubuntu 16.04

Airflow is installed on:

home/ubuntu/airflow

I have path of:

etc/systemd

The docs says to:

Copy (or link) them to /usr/lib/systemd/system

  1. Copy which of the files?

copy the airflow.conf to /etc/tmpfiles.d/

  1. What is tmpfiles.d ?

  2. What is # AIRFLOW_CONFIG= in the airflow file?

Or in another words... a more "down to earth" guide on how to do it?

airflow
2 Answers

Integrating Airflow with systemd files makes watching your daemons easy as systemd can take care of restarting a daemon on failure. This also enables to automatically start airflow webserver and scheduler on system start.

Edit the airflow file from systemd folder in Airflow Github as per the current configuration to set the environment variables for AIRFLOW_CONFIG, AIRFLOW_HOME & SCHEDULER.

Copy the services files (the files with .service extension) to /usr/lib/systemd/system in the VM.

Copy the airflow.conf file to /etc/tmpfiles.d/ or /usr/lib/tmpfiles.d/. Copying airflow.conf ensures /run/airflow is created with the right owner and permissions (0755 airflow airflow). Check whether /run/airflow exist with airflow:airflow owned by airflow user and airflow group if it doesn't create /run/airflowfolder with those permissions.

Enable this services by issuing systemctl enable <service> on command line as shown below.

sudo systemctl enable airflow-webserver
sudo systemctl enable airflow-scheduler

airflow-scheduler.service file should be as below:

[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
EnvironmentFile=/etc/sysconfig/airflow
User=airflow
Group=airflow
Type=simple
ExecStart=/bin/airflow scheduler
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target
user_56915250
user_48875300
Related