enable_tcp_keepalive in Airflow version 1.10.9

Himanshu Malhotra 0
Viewed 25

I am using Airflow 1.10.9 via AKS ( Azure ) Kubernetes.

The issue that I am facing is that if the pod status gets terminated completed, the job is still showing as Running on airflow.

The solution I found on the web is to upgrade the Airflow to 2.0 + version, however due to major code changes in the production environment, I am unable to upgrade the Airflow version.

The details about the solution is available at this Link It says that I need to have enable_tcp_keepalive = True

Is there a way that I can implement this in Airflow 1.10.9 ? Or are there any other solutions to resolve this ?

Below are the screenshot for references.

enter image description here

enter image description here

kubernetes airflow
1 Answers

In newer versions of Airflow, there is a _enable_tcp_keepalive function.

You can copy this function definition reference above over to your airflow init script (airflow_local_settings.py). You should of course clean up the function to your taste.

Then, patch the get_kube_client function in your version such that it calls this function.

# file:airflow_local_settings.py

from airflow import kubernetes

def _patch_get_kube_client():
    orig_get_kube_client = kubernetes.get_kube_client
    if kubernetes.get_kube_client.__name__ == 'patch_get_kube_client':
        return
    
    def patch_get_kube_client(*args, **kwargs):
        _enable_tcp_keepalive()
        return orig_get_kube_client(*args, **kwargs)
    
    kubernetes.get_kube_client = patch_get_kube_client

_patch_get_kube_client()
Oluwafemi Sule0
Related