I am creating a composer from terraform where I want to pass a json as input variable
Terraform code:
software_config{
env_variables{
AIRFLOW_VAR_MYJSON ="{'__comment1__': 'This the global section', 'project_id':'testproject', 'gce_zone':'us-east1-c', 'gce_region':'us-east1','networkname':'vpc1', 'subnetwork':'https://www.googleapis.com/compute/v1/projects/testproject/regions/us-east1/subnetworks/subnet1'}"
}
}
I am trying to read the value of AIRFLOW_VAR_MYJSON in DAG , but it is not working as the value is not recognized as JSON. I tried converting it and then deserializing it with following code:
JSONList = Variable.get("MYJSON")
jsonvar = json.dumps(JSONList)
setting_var = Variable.set("settings", jsonvar)
dag_config = Variable.get("settings", deserialize_json=True)
but it is not working.
I have also tried using
dag_config =json.loads(jsonvar)
then reading value as
project_id = dag_config["project_id"]
but I get error : "string indices must be integers"
Please suggest a way to resolve this.
NOTE : I know the gcloud command to set variables from json file but that is not working in my case as the project is in VPC and kubernetes clusters are giving timeout or handshake error, so I have ruled out use of this option