We have a spark job written in spark 3.0.3, now we are migrating the job to spark 3.2.2 . After migration we see below issue
Generated event log size is around 2GB earlier it was 500MB(in Spark 3.0.3) Job took double time to run with the same resource. ( earlier it took 8 mins and now with 3.2.2 version took 16mins approx.) See new stages are created in spark 3.2.2 version. Is there any new parameter we have to configure for spark 3.2.2 version??
Another problem , Because of the huge eventlog size, we are unable to open history server UI for this particular job. getting
URI: /history/spark-b4e49f2681f7407aa7182b241025e5b0/jobs/ STATUS: 500 MESSAGE: org.sparkproject.guava.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space SERVLET: org.apache.spark.deploy.history.HistoryServer$$anon$1-5bbbdd4b CAUSED BY: org.sparkproject.guava.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space CAUSED BY: java.lang.OutOfMemoryError: Java heap space
Also we can see new stages are getting generated in spark 3.2.2 compared to spark 3.0.3 version.