How can I deploy airflow on Kubernetes "out of the box"?

user_11271048 0
Viewed 608

I am somewhat new to Kubernetes, and I am trying to learn about deploying airflow to Kubernetes.

My objective is to try to deploy an "out-of-the-box" (or at least closer to that) deployment for airflow on Kubernetes. I have created the Kubernetes cluster via Terraform (on EKS), and would like to deploy airflow to the cluster. I found that Helm can help me deploy airflow easier relative to other solutions.

Here is what I have tried so far (snippet and not complete code):

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  load_config_file       = false
}


provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

data "helm_repository" "airflow" {
  name = "airflow"
  url  = "https://airflow-helm.github.io/charts"
}


resource "helm_release" "airflow" {
  name  = "airflow-helm"
  repository = data.helm_repository.airflow.metadata[0].name
  chart = "airflow-chart"
}

I am not necessarily fixed on using Terraform (I just thought it might be easier and wanted to keep state). So I am also happy to discover other solutions that will help me airflow with all the pods needed.

airflow kubernetes
1 Answers

You can install it using Helm from official repository, but there are a lot of additional configuration to consider. The Airflow config is described in chart's values.yaml. You can take a look on this article to check example configuration.

For installation using terraform you can take a look into this article, where both Terraform config and helm chart's values are described in detail.

user_122377320
Related