wildcard search in FileSensor airflow

user_12503858 0
Viewed 1383

I have a file path as 

fpath = "/path/abc20200216"

where date after abc keeps changing every month.

Now I want to use this in FileSensor

file_sensor = FileSensor(task_id= "file_sensor_job", poke_interval= 30, filepath = fpath, run_as_user='xyzzy',dag=dag)

how can I pass a wildcard keyword in the Operator.

Or is there any other alternative for sensing file in remote server using airflow?

airflow shell python
1 Answers

As of this PR You could pass a file blob to filepath for FileSensor.

For eg:

prefix = 'abc'

fileblob = os.path.join(os.path.dirname('/path/'), prefix +'*')

file_sensor = FileSensor(task_id= "file_sensor_job", poke_interval= 30, filepath = fileblob, run_as_user='xyzzy',dag=dag)

You could also take a look this test case from Airflow repo

user_71560100
Related