Airflow FileSensor taks is not working,The file path is coming out weird

user_18566392 0
Viewed 199

Airflow FileSensor taks is not working, alwasy in running status. I created a file connection as follows.

Connection Id: file_sensor Connection Type: File(path) Extra: {"path": "/opt/airflow/sensor"}

I don't know why AIRFLOW_CTX_DAG_OWNER appears as ****.

*** Reading local file: /opt/airflow/logs/sensor_test/sensor_a/2022-03-24T07:52:07.069026+00:00/1.log
[2022-03-24, 16:52:08 KST] {taskinstance.py:1037} INFO - Dependencies all met for <TaskInstance: sensor_test.sensor_a manual__2022-03-24T07:52:07.069026+00:00 [queued]>
[2022-03-24, 16:52:08 KST] {taskinstance.py:1037} INFO - Dependencies all met for <TaskInstance: sensor_test.sensor_a manual__2022-03-24T07:52:07.069026+00:00 [queued]>
[2022-03-24, 16:52:08 KST] {taskinstance.py:1243} INFO - 
--------------------------------------------------------------------------------
[2022-03-24, 16:52:08 KST] {taskinstance.py:1244} INFO - Starting attempt 1 of 1
[2022-03-24, 16:52:08 KST] {taskinstance.py:1245} INFO - 
--------------------------------------------------------------------------------
[2022-03-24, 16:52:08 KST] {taskinstance.py:1264} INFO - Executing <Task(FileSensor): sensor_a> on 2022-03-24 07:52:07.069026+00:00
[2022-03-24, 16:52:08 KST] {standard_task_runner.py:52} INFO - Started process 960 to run task
[2022-03-24, 16:52:08 KST] {standard_task_runner.py:76} INFO - Running: ['***', 'tasks', 'run', 'sensor_test', 'sensor_a', 'manual__2022-03-24T07:52:07.069026+00:00', '--job-id', '349', '--raw', '--subdir', 'DAGS_FOLDER/***SensorTest.py', '--cfg-path', '/tmp/tmp5wiingk3', '--error-file', '/tmp/tmplhcg37hy']
[2022-03-24, 16:52:08 KST] {standard_task_runner.py:77} INFO - Job 349: Subtask sensor_a
[2022-03-24, 16:52:08 KST] {logging_mixin.py:109} INFO - Running <TaskInstance: sensor_test.sensor_a manual__2022-03-24T07:52:07.069026+00:00 [running]> on host b05251258af1
[2022-03-24, 16:52:08 KST] {taskinstance.py:1431} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=***
AIRFLOW_CTX_DAG_ID=sensor_test
AIRFLOW_CTX_TASK_ID=sensor_a
AIRFLOW_CTX_EXECUTION_DATE=2022-03-24T07:52:07.069026+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-03-24T07:52:07.069026+00:00
[2022-03-24, 16:52:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:53:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:54:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:55:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:56:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:57:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:58:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 16:59:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:00:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:01:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:02:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:03:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:04:08 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:05:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:06:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:07:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:08:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:09:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:10:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:11:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:12:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:13:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:14:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:15:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:16:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:17:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:18:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:19:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:20:09 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:21:10 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:22:10 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:23:10 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:24:10 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
[2022-03-24, 17:25:10 KST] {filesystem.py:59} INFO - Poking for file /opt/***/sensor/a.txt
airflow-scheduler airflow python
1 Answers

The *** is just masking as explained in the docs.

if you have a connection with a password of a, then every instance of the letter a in your logs will be replaced with ***. To disable masking you can set hide_sensitive_var_conn_fields to false.

The proper way to handle this is simply don't use airflow as password. You should use more complex password - it will give you more protection and also will show the actual file path in the logs.

As for why it's poking - probably this means that the file is not present.

user_146244090
Related