Airflow - Why is the externalDatabase configuration breaking helm upgrade?

user_4526177 0
Viewed 926

I am trying to deploy Airflow using Helm charts, for a personal POC, but I have been facing some issues with the deployment and could not find clear instructions to solve my issue - which is why I am seeking help here.

Context around the issue

A bit of background of the POC first - I want to deploy a K8S cluster that hosts airflow, connect it to a git repo that hosts the dags and has the metastore and cache hosted externally from K8S Airflow.

I have successfully deployed Airflow to a local Kubernetes cluster using kind and Airflow's default helm chart. On the helm chart I have specified that the executor mode to be used must be KubernetesExecutor.

I have also configured Airflow to sync up the DAGs to/from a bitbucket repository.

The issue and current implementation

I am having issues with connecting Airflow with the external services - I have created an Azure PostgreSQL server, created a airflow database, created a admin user on the psql as follows:

CREATE DATABASE airflow;
CREATE USER aflw_admin WITH PASSWORD 'some_password';
GRANT ALL PRIVILEGES ON DATABASE airflow TO aflw_admin;
ALTER USER aflw_admin SET search_path = public;

Since I am using helm to deploy, I have my values.yaml as follows:

postgresql:
  enabled: false 

externalDatabase:
  type: postgres
  host: dbname.postgres.database.azure.com
  port: 5432
  database: airflow
  user: aflw_admin
  passwordSecretKey: "postgresql-password"

data:
  metadataSecretName: ~
  resultBackendSecretName: ~

  metadataConnection:
    user: aflw_admin 
    pass:  some_password
    protocol: postgresql
    host: dbname.postgres.database.azure.com
    port: 5432
    db: airflow 
    sslmode: require 
    
  resultBackendConnection:
    user: aflw_admin 
    pass:  some_password
    protocol: postgresql
    host: dbname.postgres.database.azure.com
    port: 5432
    db: airflow 
    sslmode: require

The secret postgresql-password was created by the following:

kubectl create secret generic airflow-postgresql --from-literal=postgresql-password=$(openssl rand -base64 13) --namespace airflow

I deployed the solution using:

kubectl apply -f ./helm/variables.yaml
helm upgrade --install airflow apache-airflow/airflow -n airflow -f ./values.yaml --debug

What I have tried and problem details

After some back and forward, I figured out that by reverting the configuration - aka, setting postgresql enabled to true and removing the metadataConnection, resultBackendConnection and externalDatabase sections from the values.yaml file - I could deploy the postgres service successfully but with the tradeoff that the postgresql is NOT a external service, which helps to at least partially isolate the problem.

So, if I go back to the initial configuration and try to deploy it, the results I get are :

  • first I get a timeout --> in order to face this issue, I naturally increased the timeout duration to a bigger value like 20m0s;
  • after I increase the timeout I get an error BackoffLimitExceeded and nothing is deployed.

Here's a log of the helm deployment in question:

history.go:56: [debug] getting history for release airflow
upgrade.go:142: [debug] preparing upgrade for airflow
upgrade.go:150: [debug] performing update for airflow
upgrade.go:322: [debug] creating upgraded release for airflow
client.go:218: [debug] checking 20 resources for changes
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-create-user-job"
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-migrate-database-job"
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-scheduler"
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-statsd"
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-triggerer"
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-webserver"
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-worker"
client.go:501: [debug] Looks like there are no changes for Secret "airflow-airflow-metadata"
client.go:501: [debug] Looks like there are no changes for Secret "airflow-webserver-secret-key"
client.go:501: [debug] Looks like there are no changes for ConfigMap "airflow-airflow-config"
client.go:501: [debug] Looks like there are no changes for Role "airflow-pod-launcher-role"
client.go:501: [debug] Looks like there are no changes for Role "airflow-pod-log-reader-role"
client.go:501: [debug] Looks like there are no changes for RoleBinding "airflow-pod-launcher-rolebinding"
client.go:501: [debug] Looks like there are no changes for RoleBinding "airflow-pod-log-reader-rolebinding"
client.go:501: [debug] Looks like there are no changes for Service "airflow-statsd"
client.go:501: [debug] Looks like there are no changes for Service "airflow-webserver"
client.go:510: [debug] Patch Deployment "airflow-scheduler" in namespace airflow
client.go:510: [debug] Patch Deployment "airflow-statsd" in namespace airflow
client.go:510: [debug] Patch Deployment "airflow-triggerer" in namespace airflow
client.go:510: [debug] Patch Deployment "airflow-webserver" in namespace airflow
client.go:267: [debug] Deleting Secret "airflow-postgresql" in namespace airflow...
client.go:270: [debug] Unable to get obj "airflow-postgresql", err: secrets "airflow-postgresql" not found
client.go:267: [debug] Deleting Service "airflow-postgresql-headless" in namespace airflow...
client.go:270: [debug] Unable to get obj "airflow-postgresql-headless", err: services "airflow-postgresql-headless" not found
client.go:267: [debug] Deleting Service "airflow-postgresql" in namespace airflow...
client.go:270: [debug] Unable to get obj "airflow-postgresql", err: services "airflow-postgresql" not found
client.go:267: [debug] Deleting StatefulSet "airflow-postgresql" in namespace airflow...
client.go:270: [debug] Unable to get obj "airflow-postgresql", err: statefulsets.apps "airflow-postgresql" not found
client.go:299: [debug] Starting delete for "airflow-run-airflow-migrations" Job
client.go:128: [debug] creating 1 resource(s)
client.go:529: [debug] Watching for changes to Job airflow-run-airflow-migrations with timeout of 20m0s
client.go:557: [debug] Add/Modify event for airflow-run-airflow-migrations: ADDED
client.go:596: [debug] airflow-run-airflow-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:557: [debug] Add/Modify event for airflow-run-airflow-migrations: MODIFIED
client.go:596: [debug] airflow-run-airflow-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:557: [debug] Add/Modify event for airflow-run-airflow-migrations: MODIFIED
upgrade.go:433: [debug] warning: Upgrade "airflow" failed: post-upgrade hooks failed: job failed: BackoffLimitExceeded
Error: UPGRADE FAILED: post-upgrade hooks failed: job failed: BackoffLimitExceeded
helm.go:84: [debug] post-upgrade hooks failed: job failed: BackoffLimitExceeded
UPGRADE FAILED
main.newUpgradeCmd.func2
        helm.sh/helm/v3/cmd/helm/upgrade.go:199
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.3.0/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.3.0/command.go:974
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.3.0/command.go:902
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
        runtime/proc.go:255
runtime.goexit
        runtime/asm_amd64.s:1581
make: *** [Makefile:46: deploy-airflow] Error 1

This behavior leads me to think that it is some sort of configuration error, but I can't pinpoint what.

What can I be misconfiguring in my helm chart that could have broken the helm upgrade?

The versions for helm/airflow/psql are below:

airflow kubernetes-helm azure postgresql
1 Answers

It's sometimes difficult to diagnose such problems because there are so many moving parts. Although, I have set up Airflow on Azure AKS (Postgres sslmode: require) and AWS EKS (RDS Postgres sslmode: disable), each with their own issues.

Perhaps remove config for externalDatabase and resultBackendConnection. Why? - because resultBackendConnection will use metadataConnection if it is not configured. I haven't seen the externalDatabase key in my current config file v2.2.4. Are you using -f values.yaml to override helm install with the correct values.yaml?

If you disable postgresql

postgresql:
  enabled: false

as you have, then you need to configure metadataConnection for your external database.

I only configured the metadataSecretName once I had the connection to Postgres working properly.

Also, try sslmode: disable in the metadataConnection config.

Once I got the config file the way I wanted it, I un-installed Airflow then re-installed:

  1. helm delete airflow
  2. dropped airflow db and re-created it
  3. kubectl delete secrets [airflow-xxxxx-xxxxx] All the secrets in the namespace, because the db migration didn't work properly inbetween the errors.
  4. kubectl delete pvc (then made sure the pv's were also deleted)

After that I re-installed and it was all good, which is not a lot of work, but ensures I can re-deploy with the correct values.

Oh, and do remember to set up PSQL and make sure you can actually connect from your command line as an extra check.

user_104074480
Related