Too many transactions to Azure Fileshare from airflow

user_10388791 0
Viewed 163

I have a question regarding airflow when running in AKS.

We have deployed airflow in AKS and have mounted Azure Fileshare to the airflow pods. We have using this fileshare for DAG Folder. However, there are huge transactions (at least 20K for every 5 minutes) from airflow to the FileShare which is incurring us lot of costs from Azure. FYI - billing on Azure fileshare is done based on the number of transactions and not the amount of size that we use.

Could you please let me know if this is an issue with having DAG folder in Fileshare? If so, is there any other approach that we can use. Have tried mounting managed disk (instead of fileshare) to the pods. But issue with managed disk is that we cant mount a disk to more than 1 pod.

Appreciate for the help/ideas.

Thanks

airflow-scheduler azure-aks airflow
1 Answers

I was running into a similar issue, having 8k transactions every 5 minutes for just 3 DAGs. I got it down to about 800 transactions every 5 minutes by setting file_parsing_sort_mode to alphabetical.

https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#file-parsing-sort-mode

The default setting for this, which is modified_time would make the DAG processor retrieve the last modified time of the file from the fileshare on every loop. Weirdly, this action even triggers write operations which are more costly than read operations.

https://github.com/apache/airflow/blob/2d79d730d7ff9d2c10a2e99a4e728eb831194a97/airflow/dag_processing/manager.py#L982-L1008

user_66546200
Related