Cannot setup a MySQL Backend for Airflow LocalExecutor

user_17131407 0
Viewed 213

I need to run dags in parallel but do not need significant scaling, so LocalExecutor can do the job just fine. I looked through the Airflow docs and first created a MySQL database:

CREATE DATABASE airflow_db CHARACTER SET utf8;
CREATE USER <user> IDENTIFIED BY <pass>;
GRANT ALL PRIVILEGES ON airflow_db.* TO <user>;

I then modify the following parameters in the airflow.cfg file:

executor = LocalExecutor
sql_alchemy_conn = mysql+mysqlconnector://<user>:<pass>@localhost:3306/airflow_db

When I run airflow db init, I run into the following error message:

AttributeError: 'MySQLConverter' object has no attribute '_dagruntype_to_mysql'

During handling of the above exception, another exception occurred: TypeError: Python 'dagruntype' cannot be converted to a MySQL type

Please note that nothing else in the airflow.cfg file was altered and that using the default SequentialExecutor with sqlite lets everything run just fine. Also note that I am using Airflow version 2.2.0

airflow-2.x airflow backend
1 Answers

I found the solution to my own question. Instead of using the mysqlconnector, I used the pymysql driver:

pip install PyMySQL

The airflow.cfg parameters can then be adjusted as follows:

sql_alchemy_conn = mysql+pymysql://<user>:<pass>@localhost:3306/airflow_db

All else can stay the same.

user_171314070
Related