Purpose of spark worker directories in /work/app-xxxxxxx/{0, 1, 2, ...} and periodic cleanup

user_4414336 0
Viewed 1117

I'm running a Spark 3.4 long running structured streaming job. Whenever the job starts, an application directory of the form app-xxxxxxxxxx is created for the job in the work directory. However within that directory, additional directories are created, the first being named 0, the second named 1 and so on.

My first question is, why are these directories being created? Over the course of the structured streaming job, the micro batch may get triggered 20 times but only 4 of these sub directories under the app-xxxxxxxxxx directory are created, the point being that the creation of these sub directories doesn't correspond to execution of a micro batch. So, I'm not sure why they're being created.

My second related question is, how can I configure Spark to delete these folders after a certain amount of time? Each contains the application .jar and stderr and stdout files, so over time they take up a significant amount of space. My understanding is that setting spark.worker.cleanup.enabled=true only enables cleanup for stopped applications. However in my case, I have a long running application which I would like to enable cleanup for.

spark-streaming apache-spark
1 Answers

You are talking about the work directory and the configuration spark.worker, so my assumption is you are running the streaming job in Spark's standalone mode (not using a cluster manager such as YARN because things are quite different there).

According to the documentation on Spark Standalone Mode the work directory is described as: Directory to run applications in, which will include both logs and scratch space (default: SPARK_HOME/work).

Here scratch space means that it is "including map output files and RDDs that get stored on disk. This should be on a fast, local disk in your system. It can also be a comma-separated list of multiple directories on different disks."

In the work folder you will find for each application the .jar libraries such that the executor have access to the libraries. In addition, it contains some temporary data based on the processing logic and actual data (not on the amount of processing triggers). The sub-folders 0, 1 are incremental for different jobs/stages or runs of the same application. (To be frank, I am not fully knowledgeable about those sub-folders.)

The cleaning of this folder can be adjusted by the following three configurations for the SPARK_WORKER_OPTS as described here:

spark.worker.cleanup.enabled - Default: false: Enable periodic cleanup of worker / application directories. Note that this only affects standalone mode, as YARN works differently. Only the directories of stopped applications are cleaned up. This should be enabled if spark.shuffle.service.db.enabled is "true"

spark.worker.cleanup.interval - Default: 1800 (30 minutes): Controls the interval, in seconds, at which the worker cleans up old application work dirs on the local machine.

spark.worker.cleanup.appDataTtl - Default: 604800 (7 days, 7 * 24 * 3600): The number of seconds to retain application work directories on each worker. This is a Time To Live and should depend on the amount of available disk space you have. Application logs and jars are downloaded to each application work dir. Over time, the work dirs can quickly fill up disk space, especially if you run jobs very frequently.

user_122089100
Related