Airflow - Copy blob from/to Google Cloud Storage within 2 different projects

user_4800907 0
Viewed 534

I'm trying to copy a blob from the GCS bucket A in the project X to the bucket B in the project Y using Airflow.

It seems that the available operator (GCSToGCSOperator) works well only between two buckets within the same project.

How can I have achieve the copy in my case?

I'd like to avoid using a BashOperator...

Thanks!!

google-cloud-composer airflow google-cloud-storage google-cloud-platform
1 Answers

Option 1: Using CloudDataTransferServiceCreateJobOperator which creates a transfer job using Google API. You can find information about it in docs. Note that this require the service account to have access to both. If this is not the case then it's not supported yet See Use Google Storage Transfer API to transfer data from external GCS into my GCS

Option 2: Use GCSToLocalFilesystemOperator with project 1 and then LocalFilesystemToGCSOperator using project 2.

A skeleton of this solution:

from airflow import DAG
from airflow.providers.google.cloud.transfers.local_to_gcs import LocalFilesystemToGCSOperator
from airflow.providers.google.cloud.transfers.gcs_to_local import GCSToLocalFilesystemOperator

with DAG(
    "example", schedule_interval="@daily", start_date=datetime(2021, 1, 1), catchup=False
) as dag:
    download = GCSToLocalFilesystemOperator(
        task_id="download_task",
        bucket='some_bucket',
        filename='/tmp/fake1.csv',
        object_name="test/test1.csv",
        gcp_conn_id='google_cloud_origin'
    )
    
    upload = LocalFilesystemToGCSOperator(
        task_id='upload_task',
        bucket='some_bucket',
        src='/tmp/fake1.csv',
        dst='test/test1.csv',
        gcp_conn_id='google_cloud_dest'
    )

    download >> upload

Though this is not ideal solution. It really depends on the volumes and frequency of your job. With this solution you are transferring files via the local disk - in small volumes it can be fine. This solution will work in cases of two distinct accounts as each operator is associated to a different Google connection.

user_146244090
Related