I wrote a dag, placed it in airflow/dags in a vm and when I run airflow initdb I get the following problem
(env) (base) xxxx@XXXXXX:~/airflow/dags$ airflow initdb
[2020-02-27 08:06:19,564] {settings.py:253} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=105407
DB: postgresql+psycopg2://bn_airflow%40XXXXXXX?sslmode=require
[2020-02-27 08:06:20,236] {db.py:368} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
ERROR [airflow.models.dagbag.DagBag] Bad magic number for central directory
Traceback (most recent call last):
File "/home/pgkrdatahub/.local/lib/python3.6/site-packages/airflow/models/dagbag.py", line 420, in collect_dags
safe_mode=safe_mode)
File "/home/pgkrdatahub/.local/lib/python3.6/site-packages/airflow/models/dagbag.py", line 251, in process_file
zip_file = zipfile.ZipFile(filepath)
File "/usr/lib/python3.6/zipfile.py", line 1131, in __init__
self._RealGetContents()
File "/usr/lib/python3.6/zipfile.py", line 1226, in _RealGetContents
raise BadZipFile("Bad magic number for central directory")
zipfile.BadZipFile: Bad magic number for central directory
ERROR [airflow.models.dagbag.DagBag] Bad magic number for central directory
Traceback (most recent call last):
File "/home/pgkrdatahub/.local/lib/python3.6/site-packages/airflow/models/dagbag.py", line 420, in collect_dags
safe_mode=safe_mode)
File "/home/pgkrdatahub/.local/lib/python3.6/site-packages/airflow/models/dagbag.py", line 251, in process_file
zip_file = zipfile.ZipFile(filepath)
File "/usr/lib/python3.6/zipfile.py", line 1131, in __init__
self._RealGetContents()
File "/usr/lib/python3.6/zipfile.py", line 1226, in _RealGetContents
raise BadZipFile("Bad magic number for central directory")
zipfile.BadZipFile: Bad magic number for central directory
Done.
Im not entirely sure what the issue is. I looked at the source code and I think its somehow expecting a zip file when there are none in the dags folder
dags contains
env) (base) pgkrdatahub@xxx:~/airflow/dags$ ls -al
total 32
drwxrwxr-x 5 pgkrdatahub pgkrdatahub 4096 Feb 27 08:06 .
drwxrwxr-x 4 pgkrdatahub pgkrdatahub 4096 Feb 27 04:36 ..
drwxrwxr-x 8 pgkrdatahub pgkrdatahub 4096 Feb 27 03:00 .git
-rw-rw-r-- 1 pgkrdatahub pgkrdatahub 109 Feb 27 03:00 README.md
-rw-rw-r-- 1 pgkrdatahub pgkrdatahub 5040 Feb 27 08:06 SomReviewDailyDag.py
drwxrwxr-x 2 pgkrdatahub pgkrdatahub 4096 Feb 27 08:06 __pycache__
drwxrwxr-x 8 pgkrdatahub pgkrdatahub 4096 Feb 27 06:44 sommers
Where SomReviewDailyDag imports scripts from these sommers folder.
SomReviewDailyDag looks like this
import sys
sys.path.insert(0, "/home/pgkrdatahub/airflow/dags/sommers")
from builtins import range
from airflow.operators.python_operator import PythonOperator
from airflow.operators.postgres_operator import PostgresOperator
from airflow.operators.bash_operator import BashOperator
from airflow import DAG
from datetime import datetime, timedelta
import airflow
from sommers import SomReviewJob
args = {
'owner': 'airflow',
'start_date': datetime(2019, 2, 6, 0, 0),
'email': ['xxxxx'],
'email_on_failure': True,
'email_on_retry': False
}
with DAG(dag_id = 'Sommers_Review_Daily', default_args = args, schedule_interval = '0 1 * * *', dagrun_timeout = timedelta(minutes=150)) as dag:
activate_venv = BashOperator(
task_id='Activate_Venv',
bash_command='source /pgkrdatahub/airflow/dags/sommers/env/bin/activate',
)
run_somreview_daily = PythonOperator(
task_id='SomReview_Daily',
# provide_context=True,
python_callable=SomReviewJob.main,
)
I googled and I couldn't find any reference to this issue in airflow.
I found Bad magic number error with ZipFile module in Python
which suggests that the header needs to be a certain string.
But I can't find anything else to figure out what the issue. Help would be greatly appreciated