I have installed Airflow 2.2.5 in my local VM (Oracle Virtual Box 8.6) with MySQL 8.0 as database backend and went through the installation process shown in the website (https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html).
I am using the 'sql_alchemy_conn' argument below:
mysql+mysqlconnector://airflow_user:airflow_password@localhost:3306/airflow_db
I managed to install Airflow, its dependencies (at least what has been asked throughout the process) and got to a point where I can log in to the webserver. However the webserver says the scheduler is not running, as show in the picture:
When I execute "airflow scheduler" in the terminal, I get the following error:
[2022-09-14 15:43:28,943] {authentication.py:59} INFO - package: mysql.connector.plugins
[2022-09-14 15:43:28,947] {authentication.py:60} INFO - plugin_name: mysql_native_password
[2022-09-14 15:43:28,953] {authentication.py:64} INFO - AUTHENTICATION_PLUGIN_CLASS: MySQLNativePasswordAuthPlugin
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/mysql/connector/cursor.py", line 398, in _process_params_dict
conv = to_mysql(conv)
File "/usr/local/lib/python3.6/site-packages/mysql/connector/conversion.py", line 194, in to_mysql
) from None
TypeError: Python 'taskinstancestate' cannot be converted to a MySQL type
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.6/site-packages/mysql/connector/cursor.py", line 541, in execute
stmt = _bytestr_format_dict(stmt, self._process_params_dict(params))
File "/usr/local/lib/python3.6/site-packages/mysql/connector/cursor.py", line 406, in _process_params_dict
) from err
mysql.connector.errors.ProgrammingError: Failed processing pyformat-parameters; Python 'taskinstancestate' cannot be converted to a MySQL type
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/airflow/__main__.py", line 48, in main
args.func(args)
File "/usr/local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/cli/commands/scheduler_command.py", line 75, in scheduler
_run_scheduler_job(args=args)
File "/usr/local/lib/python3.6/site-packages/airflow/cli/commands/scheduler_command.py", line 46, in _run_scheduler_job
job.run()
File "/usr/local/lib/python3.6/site-packages/airflow/jobs/base_job.py", line 242, in run
session.commit()
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1046, in commit
self.transaction.commit()
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
self._flush(objects)
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
with_traceback=exc_tb,
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
flush_context.execute()
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
uow,
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
insert,
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1136, in _emit_insert_statements
statement, params
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
distilled_params,
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.6/site-packages/mysql/connector/cursor.py", line 541, in execute
stmt = _bytestr_format_dict(stmt, self._process_params_dict(params))
File "/usr/local/lib/python3.6/site-packages/mysql/connector/cursor.py", line 406, in _process_params_dict
) from err
sqlalchemy.exc.ProgrammingError: (mysql.connector.errors.ProgrammingError) Failed processing pyformat-parameters; Python 'taskinstancestate' cannot be converted to a MySQL type
[SQL: INSERT INTO job (dag_id, state, job_type, start_date, end_date, latest_heartbeat, executor_class, hostname, unixname) VALUES (%(dag_id)s, %(state)s, %(job_type)s, %(start_date)s, %(end_date)s, %(latest_heartbeat)s, %(executor_class)s, %(hostname)s, %(unixname)s)]
[parameters: {'dag_id': None, 'state': <TaskInstanceState.RUNNING: 'running'>, 'job_type': 'SchedulerJob', 'start_date': datetime.datetime(2022, 9, 14, 18, 43, 29, 247880), 'end_date': None, 'latest_heartbeat': datetime.datetime(2022, 9, 14, 18, 43, 29, 247894), 'executor_class': 'LocalExecutor', 'hostname': 'airhost', 'unixname': 'root'}]
(Background on this error at: http://sqlalche.me/e/13/f405)
I have been clueless about this section:
"msql.connector.errors.ProgrammingError: Failed processing pyformat-parameters; Python 'taskinstancestate' cannot be converted to a MySQL type".
What does that mean? Any ideas? I haven't run any DAGs or built anything so far.