Enable Google Drive OAuth Scopes on Cloud Composer 2.0

user_17313489 0
Viewed 396

Is there a method to add/modify google oauth scopes to a Cloud Composer 2.0 environment?
When installing Composer 2.0 there is no option to modify the oauth scopes from the UI or command line. I need to add google drive to the oauth scopes on the cluster so that my existing Python Operators will be able to access sheets in a google drive.

My understanding is that Composer 2.0 installs GKE with AutoPilot mode and I have also been looking for a way to add the oauth scopes to a GKE AutoPilot cluster as well with no success. Any ideas?

Is there alternately a way to specify scopes for the Python operator execution?

google-cloud-composer google-kubernetes-engine
1 Answers

Came across this same issue and finally figured it out.

  1. Create a Google Cloud Airflow Connection
  2. Within your DAG, where you call on the operator to build the task, include the parameter:
    • gcp_conn_id = '< name of connection >'

Cloud Composer v2 doesn't support OAuth on the Environment Level anymore because it now uses GKE Autopilot. GKE Autopilot does not support OAuth since this is a legacy method of GCE. For the connections that still need OAuth you will need to define these in an Airflow Connection and use the connection in your task. The scope of the connection seems to override and preexisting scopes that may be included by default, like the full cloud-platform scope. I had to specify both spreadsheets and cloud-platform in my connection for the sheets_to_gcs operator.

user_58898740
Related