Manage connections in multi tenant Airflow instance

user_9428990 0
Viewed 364

I am running an Airflow instance on AWS ECS. Works great. However, as of recently the need for multi tenant has come up. I activated webserver:filter_by_owner and my different teams are now as intended not able to see each others DAGs. Which is great!

However, I noticed by accident that there is nothing stopping Team A from using a connectionId intended to be used by Team B.

In my example I'm running Airflow to orchestrate some SQL in Snowflake. Example:

with DAG("TEST_Dag", default_args=config.default.args, schedule_interval=None) as dag:

    t_create_table = SnowflakeOperator(
        task_id="CreateTable",
        snowflake_conn_id="CONNECTION_ID_FOR_TEAM_A",
        sql="CREATE TABLE IF NOT EXISTS TEST_TABLE (C1 INT)",
        dag=dag
    )

In this example I have setup a connection towards snowflake that I call CONNECTION_ID_FOR_TEAM_A. Each team will have their own connection id. Hence TEAM B will have a CONNECTION_ID_FOR_TEAM_B.

The problem however, lies in that there is nothing that stops Team B from using the connection of team A, which then is a great security flaw in this multi tenant Airflow setup.

Can this be solved?

airflow-scheduler airflow
1 Answers

I think there is no easy way to do this in Airflow 1.x, but there is something which can help you in Airflow 2. They added something called Secret backend. You can use it to keep you connection and to allow Airflow to fetch connections from there and not from its own database. Downside of this is that these connections will not appear in Airflow UI.

You can send user ID and use that as filter for connections on your Secret backend.

user_7591260
Related