How to solve 'no space left on device' error when running airflow on docker

user_10617728 0
Viewed 548

I'm running Airflow on docker, and the scheduler container stops running after some time with the following error

Error response from daemon: Cannot restart container airflow-scheduler_1: mkdir /var/snap/docker/common/var-lib-docker/overlay2/3355a6abc0c5f0bde0eba9026fd04a9d627562d6c6f069c4a564c5fe7738721f/merged: no space left on device

I have used

docker system prune --volumes

but this is effective for a short period of time before the container stops running again and displaying the same error.

What I'm I missing?

airflow docker ubuntu linux
1 Answers

You've got a few options:

  1. Increase the space on the docker instance itself
  2. Have a repeating script or job that prunes the volumes
  3. Map your DAGs or logs folder in the docker instance to a non-docker mount point

The third option is what we do, all our logs and code files are held outside the docker instances, so they don't get filled up unnecessarily.

We've found that the logs can sometimes be very verbose, so you might also want to look at filtering your logs or adding a log lifecycle.

user_182997500
Related