I encountered a strange NPE problem in spark these days.
Below is the piece of my code and the related exception.
Caused by: java.lang.NullPointerException
at com.iflytek.gnome.data.spark.main.SessionModelGeneratorV3$$anonfun$11.apply(SessionModelGeneratorV3.scala:231)
at com.iflytek.gnome.data.spark.main.SessionModelGeneratorV3$$anonfun$11.apply(SessionModelGeneratorV3.scala:231)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:464)
at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
We can see that NPE happens in line 231, but I've filtered null values in line 223. Code in line 231 should not throw NPE.
And the NPE just happens accidently. Most times, I just resubmit the job, and all going well.
How does this NPE happen?
Added 2021-12-28
The problem happens both in spark2.3.4 and spark2.4.4.
It seems to be related to that persist.
I removed presist from the code, and the problem never appears again.