Create airflow pools programatically

user_9731202 0
Viewed 287

I have airflow running on a docker container. I'm working with airflow version 2.0.2.

I know that I can actually create pools via the UI. But I'm looking for a way to create the pools programmatically on docker build via a pools.json file. Any way to help ?

airflow docker
1 Answers

You can do this programmatically via the REST API or CLI. The REST API allows for adding pools over HTTP one by one. The CLI has a command for importing pools from a JSON file:

airflow pools import [-h] [-v] FILEPATH

For example:

{
    "pool_1": {"slots": 5, "description": ""},
    "pool_2": {"slots": 5, "description": "test"}
}

The format is: "[pool name]": {"slots": [nr of slots], "description": "[description]"}

airflow pools import pools.json

The command works idempotent, so you can run it as many times as you like. Changes to existing pools result in an update, new pools will be created. Removing a pool from pools.json doesn't remove the pool in Airflow -- you'll have to do that manually.

Since it works idempotent, you can build a pools.json file into your Airflow image and run airflow pools import at the start of your container process, so that pools are checked at the start of every Airflow container.

user_30664280
Related