Some tasks in my Spark process (running on Amazon EMR on Ec2) stuck.
Spark history server shows
- single active job
- Stages: Succeeded/Total
- Tasks (for all stages): 1867/10521 (13 running)
- Single active stage with 13 running tasks
following stacktrace:
org.apache.spark.api.java.JavaPairRDD.repartition(JavaPairRDD.scala:121)
org.apache.beam.runners.spark.translation.GroupCombineFunctions.reshuffle(GroupCombineFunctions.java:198)
org.apache.beam.runners.spark.translation.TransformTranslator$10.evaluate(TransformTranslator.java:547)
org.apache.beam.runners.spark.translation.TransformTranslator$10.evaluate(TransformTranslator.java:529)
org.apache.beam.runners.spark.SparkRunner$Evaluator.doVisitTransform(SparkRunner.java:425)
org.apache.beam.runners.spark.SparkRunner$Evaluator.enterCompositeTransform(SparkRunner.java:370)
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:653)
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
org.apache.beam.runners.spark.SparkRunner.lambda$run$1(SparkRunner.java:224)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
edit: In logs of the executor I have found it stuck at
21/03/18 13:08:35 INFO ExternalSorter: Thread 257 spilling in-memory map of 20.6 GB to disk (10 times so far)
21/03/18 13:11:16 INFO ExternalSorter: Thread 257 spilling in-memory map of 20.6 GB to disk (11 times so far)
21/03/18 13:12:51 INFO WriteFiles: Opening writer fc922f25-637e-43ac-ab4e-be7333300ac0 for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@560c63b pane PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} destination KV{GLB, FEATURE_PROPERTY_ENTRY}
This executor has Shuffle remote reads=54Gb Shuffle read 950Mib 800,000,000 records Shuffle spill (memory)=256Gb Shuffle spill (disk)=53Gb