Installing (python3) airflow does not create airflow directory

user_8236733 0
Viewed 3142

Having problem where installing airflow (in python3) does not create the airflow directory (unlike in the python2 version, where the directory is created automatically after pip install).

[airflow@airflowetl ~]$ pip3 --no-cache-dir install apache-airflow --user
Collecting apache-airflow
  Downloading https://files.pythonhosted.org/packages/d5/12/6e5e4207483df7e82ac87079281f8f500e6610247ceb1464ed5592f4ceed/apache_airflow-1.10.5-py2.py3-none-any.whl (5.9MB)
    78% |█████████████████████████▎      | 4.6MB 1.4MB/s eta 0:00:01debug2: channel 0: window 999183 sent adjust 49393
    100% |████████████████████████████████| 5.9MB 1.4MB/s
.....
Installing collected packages: apache-airflow
Successfully installed apache-airflow-1.10.5



[airflow@airflowetl ~]$ pwd
/home/airflow
[airflow@airflowetl ~]$ ls
Documents  Downloads  projects  tmp

Any idea what is happening here and how to fix?

airflow
1 Answers

Found that unlike with python2, airflow in python3 does not create the ~/airflow dir until the first time airflow is run

The first time you run Airflow, it will create a file called airflow.cfg in your $AIRFLOW_HOME directory (~/airflow by default). https://airflow.readthedocs.io/en/1.9.0/configuration.html#setting-configuration-options

I am assuming this means any airflow ... CLI command, but in my case I just ran the next installation step (though I assume something like airflow list_dags would also work)...

[airflow@airflowetl ~]$ airflow initdb
[2019-10-18 13:36:56,916] {__init__.py:51} INFO - Using executor SequentialExecutor
DB: sqlite:////home/airflow/airflow/airflow.db
[2019-10-18 13:36:57,438] {db.py:369} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> e3a246e0dc1, current schema
INFO  [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
/home/airflow/.local/lib/python3.6/site-packages/alembic/ddl/sqlite.py:39: UserWarning: Skipping unsupported ALTER for creation of implicit constraint
  "Skipping unsupported ALTER for "
....
INFO  [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 004c1210f153, increase queue name size limit
WARNI [airflow.utils.log.logging_mixin.LoggingMixin] cryptography not found - values will not be stored encrypted.
Done.
[airflow@airflowetl ~]$ cd
[airflow@airflowetl ~]$ ls
airflow
[airflow@airflowetl ~]$ tree airflow/
airflow/
├── airflow.cfg
├── airflow.db
├── logs
│   └── scheduler
│       ├── 2019-10-18
│       └── latest -> /home/airflow/airflow/logs/scheduler/2019-10-18
└── unittests.cfg

4 directories, 3 files
[airflow@airflowetl ~]$ mkdir airflow/dags

* (For others having airflow install problems) Note also that the airflow CLI binary location was not initially in my path; had to do a quick

[airflow@airflowetl ~]$ cd
[airflow@airflowetl ~]$ pwd
/home/airflow
[airflow@airflowetl ~]$ find . -name airflow
./.local/lib/python3.6/site-packages/airflow
./.local/lib/python3.6/site-packages/airflow/bin/airflow
./.local/lib/python3.6/site-packages/airflow/www/templates/airflow
./.local/lib/python3.6/site-packages/airflow/www_rbac/templates/airflow
./.local/bin/airflow
[airflow@airflowetl ~]$ echo "PATH=$PATH:~/.local/bin" >> ~/.bashrc
[airflow@airflowetl ~]$ source ~/.bashrc

before could use the binary (another problem I did not have when installing airflow for python2).

user_82367330
Related