How to check if folder exists in Google Cloud Storage using airflow operator

user_11731607 0
Viewed 1459

I am trying to build a DAG that first checks if a given path in the Google Cloud Storage exists or not. if not , it will simply exist/end to DAG. is there any Airflow operator available to do the same?

airflow google-cloud-storage google-cloud-platform
2 Answers

Easy answer: The folder doesn't exist!!

More seriously, the folder really doesn't exist, all the object are stored at the root (the bucket). The folder is only a human representation to easily classify and navigate through the files.

However, you can achieve what you want: perform a list object operation and provide a prefix. The prefix of the file can contain / which is the human representation of the folder.

I found this operator, but I'm not a Composer user, it may be the wrong version, but the idea is this one: List the file with the targeted prefix. If no file, no folder!

user_113725930

As per this documentation of Airflow operators for Google Cloud Storage, there is no direct operator for detecting GCS paths. However, we can check if the bucket/object (which constitutes the path) exists or not. If the bucket/object doesn't exist, there won't be a path for that object.

You can try using GCSListObjectsOperator to list the objects in a particular bucket by adding appropriate prefixes.

For example:

with models.DAG(
    "example_gcs_sensors",
    start_date=days_ago(1),
    schedule_interval=None,
    tags=['example'],
) as dag2:

    GCS_Files = GCSListObjectsOperator(
    task_id='GCS_Files',
    bucket=BUCKET_1,
    prefix='hi-',
 )
    
    delete_bucket = GCSDeleteBucketOperator(task_id="delete_bucket", bucket_name=BUCKET_1)

    GCS_Files >> delete_bucket

Here GCSListObjectsOperator will list all the objects in BUCKET_1, if the object with the specified prefix is not present, ‘GCS_Files’ task will fail.

Alternatively, Airflow also provides GCS sensors: GCSObjectExistenceSensor and GCSObjectsWithPrefixExistenceSensor operator which can be used to check the existence of objects in a bucket.

user_157504730
Related