I am completely new to apache airflow and trying to use fresh airflow 2.1.0 correctly. Webserver and scheduler are working fine (I suppose. Cause I can list dags, I can interact with UI). If it`s important, I run airflow under docker container centos 7 based.
I`ve stuck with importing dags.
I have installed required modules, but have errors.
Errors:
Broken DAG: [/opt/airflow/dags/GWS_churn_prediction_load_data/gws_main_dag.py] Traceback (most recent call last): File "/opt/airflow/dags/GWS_churn_prediction_load_data/gws_main_dag.py", line 11, in <module> from airflow.operators.postgres_operator import PostgresOperator File "/usr/local/lib/python3.6/site-packages/airflow/operators/postgres_operator.py", line 23, in <module> from airflow.providers.postgres.operators.postgres import Mapping, PostgresOperator # noqa ModuleNotFoundError: No module named 'airflow.providers.postgres'
Broken DAG: [/opt/airflow/dags/EDW/EDW_000003/test.py] Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/opt/airflow/dags/EDW/EDW_000003/test.py", line 7, in <module> df2 = df.withColumn("AUDIT_RK",col("AUDIT_ID").cast("Integer")+1)#.show(false) NameError: name 'col' is not defined
Broken DAG: [/opt/airflow/dags/EDW/EDW_000003/EDW_000003_AUDIMEX.py] Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/opt/airflow/dags/EDW/EDW_000003/EDW_000003_AUDIMEX.py", line 3, in <module> from airflow.contrib.example_dags.libs.helper import print_stuff ModuleNotFoundError: No module named 'airflow.contrib.example_dags'
Broken DAG: [/opt/airflow/dags/GWS_churn_prediction_model/gws_model_main_dag.py] Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 179, in apply_defaults result = func(self, *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 511, in __init__ "arguments were:\n**kwargs: {k}".format(c=self.__class__.__name__, k=kwargs, t=task_id), airflow.exceptions.AirflowException: Invalid arguments were passed to PythonSensor (task_id: check_new_week). Invalid arguments were: **kwargs: {'provide_context': True}
I installed apache-airflow-providers-postgres. This didn't help. I installed all required providers` modules (apache-airflow-providers-ssh, -apache-spark (this required for project), -ftp, -imap, -ssh and so on).
Airflow.cfg has a line with a path where I import dags: /opt/airflow/dags, but there is no dags (ofc, it`s cause dag import errors).
- Why do i see
airflow.providers.postgreserror? I've installed providers package. What is wrong? - How can i solve problem with
example_dags? What is this module exactly? I haven't find this in PyPi.
Any help and hints will be appreciated.