How to import airflow connections from a json file

Viewed 741

so I have established two connections aws_default and google_cloud_default in a json file like so

{
    "aws_default": {
        "conn_type": "s3",
        "host": null,
        "login": "sample_login",
        "password": "sample_secret_key",
        "schema": null,
        "port": null,
        "extra": null
      },
    "google_cloud_default": {
          "conn_type": "google_cloud_platform",
          "project_id": "sample-proj-id123",
          "keyfile_path": null,
          "keyfile_json": {sample_json},
          "scopes": "sample_scope",
          "number_of_retries": 5,
        } 
}

I have a local airflow server containerized in docker. What I am trying to do, is to import the connections from this file, that way I don't need to define the connections in the Airflow UI.

I have an entrypoint.sh file which runs everytime the airflow image is built.

I have included this line airflow connections import connections.json in that shell file.

in my docker-compse.yaml file, I have added a binded a volume like so

          - type: bind
          source: ${HOME}/connections.json
          target: /usr/local/airflow/connections.json

However, when I run my DAG locally, which includes hooks that connect to these connections, I receive errors: i.e.

The conn_id `google_cloud_default` isn't defined

So I'm not too sure how to proceed. I was reading about Airflow's local filesystem secrets backend here

And it mentions this code chunk to establish the file path

[secrets]
backend = airflow.secrets.local_filesystem.LocalFilesystemBackend
backend_kwargs = {"variables_file_path": "/files/var.json", "connections_file_path": "/files/conn.json"}

But, as I check my airflow.cfg, I can't find this code chunk. Am I supposed to add this to airflow.cfg?

Could use some guidance here.. I know the solution is simple but I've naive to setting up a connection like this. Thanks!

0 Answers
Related