How to connect airflow[docker] to GCP?

lima 0
Viewed 40

First I connect airflow PostgresOperator and create dataset Now i want to Postgres to GCS Operator connection but Always see me invalid dsn: invalid connection option "extra__google_cloud_platform__key_path"

how to do I change??

my airflow connection enter image description here

and my Dags

postgres_to_gcs_task = PostgresToGCSOperator(
    task_id = 'postgres_to_gcs_task',
    postgres_conn_id='postgres_to_gcs',
    sql = 'SELECT * FROM game;',
    bucket= 'lol-airflow-bucket',
    filename= 'lol-airflow-bucket/my_game/1',
    export_format= 'csv',
    dag=dag
)
airflow google-cloud-platform python
1 Answers

I think there is an issue with the json key or the key file path.

Can you confirm the path and the json key are corrects ?

This link from official documentation explains how to create a connection in Airflow : https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html

You can also check this interesting post on this topic :

https://junjiejiang94.medium.com/get-started-with-airflow-google-cloud-platform-docker-a21c46e0f797

Another direction if it can help : If you want to use a managed service on Google Cloud with Airflow, you can create a Cloud Composer cluster (V2 is better)

In this case, no need to care about the connection on GCP, you only need to give a Service account with the expected permissions to the Cloud Composer configuration when creating the cluster.

Mazlum Tosun0
Related