I am configuring Airflow with CeleryExecutor with 2 RHEL machines on GCP. We can consider one as master and other as client.
I have mounted master directory /airflow on client and configured airflow.cfg file on master with required changes like replacing localhost even in sqlAlchemy connection.
Created rabbitmq queues on both master as well as client (configured rabbitmq cluster also). Once all the setup is ready, started services web server, worker and scheduler on master.
All services starts up perfectly. I am able to perform airflow initdb as well.
But, After some seconds first scheduler goes down with MySQL connection error and then gradually worker and web server errors out and shuts down.
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2006, "Can't connect to MySQL server on '<MASTER IP>' (111)")
{celery_executor.py:264} ERROR - Error fetching Celery task state, ignoring it:OperationalError('(_mysql_exceptions.OperationalError) (2006, "Can\'t connect to MySQL server on <MASTERIP>' (111)")',)
Other error observed in log:
{celery_executor.py:264} ERROR - Error fetching Celery task state, ignoring it:
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server during query')
Master is the one where MySQL database in installed.
After the initial setup when i had created user on MySQL, tested logins with the user which all are successful from localhost as well as from client machine. Not, sure despite all access available to mysql user and database why above error is received.
If anyone came across the similar issue while configuring Airflow in distributed mode using celeryExecutor, kindly share the workaround.
It would be great help.
Note: I have tried out other solutions like editing /etc/my.cnf with bind-address parameter, user and host level privileges but no success.
Thanks in advance.