Spark streaming job doesn't delete shuffle files

user_2128732 0
Viewed 1173

I have a spark streaming job that run on EMR, read messages from Kafka and output to S3.

I use emr-5.17.0, i.e hadoop 2.8.4, spark 2.3.1

The problem is that shuffle files are being accumulated in: /mnt/yarn/usercache/hadoop/appcache/application_1540126328960_0001/

and never get deleted until I run out of space on disk

The files looks like: shuffle_328_127_0.index, shuffle_328_134_0.data

I did try to update Yarn's policy like so: yarn.nodemanager.localizer.cache.cleanup.interval-ms 300000 yarn.nodemanager.localizer.cache.target-size-mb 5000

But it did not solve the problem.

Currently I restart the job automatically and start a new one every few hours and when the application is stopped it deletes the cache

What can I do in order to make yarn delete the cache files?

Thanks

spark-streaming apache-kafka apache-spark
2 Answers

I had a cron job (hourly) clean up the files/directories older than 6 hours to fix the disk out of space issue, i did not find a parameter in spark/yarn that would do this automatically, here are the details.

crontab entry.

0 * * * * /home/hadoop/clean_appcache.sh >/dev/null 2>&1

clean_appcache.sh

#!/bin/bash

BASE_LOC=/mnt/yarn/usercache/hadoop/appcache
sudo find $BASE_LOC/ -mmin +360 -exec rmdir {} \;
sudo find $BASE_LOC/ -mmin +360 -exec rm {} \;
user_26774850

Disable dynamic allocation. Executors are killed before doing all the cleanup. If the executors are not constantly killed/created the leak is gone.

To do that, in your spark-submit command:

  --conf "spark.dynamicAllocation.enabled=false"
user_38647280
Related