I'm running a glue job on a s3 dataset of around 6 million files totaling to about 80 GB, on which I am performing a window function and writing to a different s3 location. My glue job is using 50 G2.X workers and default Spark partitioning. When I run this, I receive an error listed below. Any suggestions on how to keep from running out of storage on an executor?
scheduler.TaskSetManager (Logging.scala:logWarning(66)): Lost task 401.0 in stage 4.0 (TID 505643, 172.35.178.124, executor 13): org.apache.spark.memory.SparkOutOfMemoryError: error while calling spill() on org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@16007c9 : No space left on device at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:219) at org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:285) at org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:117) at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.acquireNewPageIfNecessary(UnsafeExternalSorter.java:383) at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.insertRecord(UnsafeExternalSorter.java:407) at org.apache.spark.sql.execution.UnsafeExternalRowSorter.insertRow(UnsafeExternalRowSorter.java:135) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.sort_addToSorter_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636) at org.apache.spark.sql.execution.window.WindowExec$$anonfun$11$$anon$1.fetchNextRow(WindowExec.scala:314) at org.apache.spark.sql.execution.window.WindowExec$$anonfun$11$$anon$1.(WindowExec.scala:323) at org.apache.spark.sql.execution.window.WindowExec$$anonfun$11.apply(WindowExec.scala:303) at org.apache.spark.sql.execution.window.WindowExec$$anonfun$11.apply(WindowExec.scala:302) at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801) at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)