so I have established two connections aws_default and google_cloud_default in a json file like so
{
"aws_default": {
"conn_type": "s3",
"host": null,
"login": "sample_login",
"password": "sample_secret_key",
"schema": null,
"port": null,
"extra": null
},
"google_cloud_default": {
"conn_type": "google_cloud_platform",
"project_id": "sample-proj-id123",
"keyfile_path": null,
"keyfile_json": {sample_json},
"scopes": "sample_scope",
"number_of_retries": 5,
}
}
I have a local airflow server containerized in docker. What I am trying to do, is to import the connections from this file, that way I don't need to define the connections in the Airflow UI.
I have an entrypoint.sh file which runs everytime the airflow image is built.
I have included this line airflow connections import connections.json in that shell file.
in my docker-compse.yaml file, I have added a binded a volume like so
- type: bind
source: ${HOME}/connections.json
target: /usr/local/airflow/connections.json
However, when I run my DAG locally, which includes hooks that connect to these connections, I receive errors: i.e.
The conn_id `google_cloud_default` isn't defined
So I'm not too sure how to proceed. I was reading about Airflow's local filesystem secrets backend here
And it mentions this code chunk to establish the file path
[secrets]
backend = airflow.secrets.local_filesystem.LocalFilesystemBackend
backend_kwargs = {"variables_file_path": "/files/var.json", "connections_file_path": "/files/conn.json"}
But, as I check my airflow.cfg, I can't find this code chunk. Am I supposed to add this to airflow.cfg?
Could use some guidance here.. I know the solution is simple but I've naive to setting up a connection like this. Thanks!