Upload gcs files to google drive using airflow

user_14236376 0
Viewed 1372

Hi I am trying to upload a file from GCS to Gdrive using

airflow.contrib.operators.gcs_to_gdrive_operator import GcsToGDriveOperator

This is how the dag looks like

copy_to_gdrive = GcsToGDriveOperator(
    task_id="copy_to_gdrive",
    source_bucket="my_source_bucket_on_gcs",
    source_object="airflow-dag-test/report.csv",
    destination_object="/airflow-test/report.csv",
    gcp_conn_id="bigquery_default",
    dag=dag
)

This code executes without any errors and in the logs I can see the file is downloaded to local successfully and uploaded to gdrive successfully as well.

This code is executed by a service account, the issue i am facing is I am not able to find the file or the directory this dag is creating uploading

I have tried several permutation/combinations of path for "destination_object" but nothing seems to work also google docs are not helpful as well.

I can see in the api logs that that the drive.create api is being called but where it is creating the file is unknown. Has anyone experienced this ? any help or tip would be of great help. Thanks!

google-drive-shared-drive airflow google-cloud-storage google-drive-api google-cloud-platform
1 Answers

Your Service account is a Google account, and, as google account, it has access to its own drive. The file are correctly copied to Drive, but to the drive of the service account!

You never specify the account, so, how Airflow can know that it has to use yours?

Look at the operator documentation

delegate_to (str) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

Use this parameter, fill it with your email and activate the domain delegation wide to your service account.

user_113725930
Related