I'm running a cluster of 15 n1-standard-v4 workers on Dataproc. My I/O data is in Avro format.
Last stage in the spark job is to save data, which finishes with StackOverflowError. The DAG is:
And worker statistics:
The log with the error:
java.lang.StackOverflowError
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:3076)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1618)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2093)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1655)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
I tried to:
investigate VM instance monitoring of failed workers (everything's ok)
balance partition size (reducing it gives the same stackoverflow error)
tune spark.executor.memoryOverhead, spark.serializer, etc. parameters
Nothing seems to work so far. The questions I want to ask are the following:
why log with error points at java.io.ObjectInputStream$BlockDataInputStream.readByte as the cause of the error?
how can I improve the investigation policy for such issues like StackOverflow?
Thanks!