I'm running a distributed Airflow(2.1.4) setup using CeleryExecutor. My main containers run at one host with 11,5 GiB memory and then I have a set of processing machines running the CeleryExecutor with 24 GiB memory. I'm running the processes using DockerOperator.
I'm doing heavy processing, where lots of data is written to mounted drive. According to docker stats The memory required by this particular container, rises steadily above 11 GiB and then the container crashes with an error that implies out of memory exception. This happens although the memory limit of the machine is twice the amount that the process uses.
I was able to run the process with this setup, if I scale down the processing a bit, so that the processing is the same, but calculation is for a smaller time frame, meaning that less memory is consumed. In addition, I'm able to run the full processing scheme if I use direct call to docker daemon instead of the Airflow DockerOperator/CeleryWorker setup. These tests tell me that this crash is not related to any error in the processing code.
If I set memlimit at the DockerOperator, I can see it at the processing machine docker stats, it doesn't have an effect on the task run though.
Question:
- Could the Mem limit of Airflow main host(11,5 GiB) somehow limit the processing on the celery host(24 GiB)?
- Is there any other mem checks that I could do to extract more information?