I am running an apache-spark job that is doing a lot of transformations described in spark ql. At some point, my job hangs. This point is when my job issues a query about dropping a table. At the same time in spark UI, I don't see that the corresponding stage is started, just a previous one is done, and a new one is not starting.
An executor's logs I see
WARN executor.Executor: Issue communicating with driver in heartbeater
...which happens exactly at the moment when drivers stop writing logs.
I dont see any issue in driver logs, but I see a warning, which happens right after some executors report that they stop trying to get in touch with driver:
19/03/25 18:28:29 WARN hdfs.DataStreamer: Exception for BP-yyy-127.0.0.1-zzz:blk_xxx
java.io.EOFException: Unexpected EOF while trying to read response from server
at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:399)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1020)
I checked the table which is about to be dropped. Nothing special, except that I see one empty parquet file there. But if I issue a drop command from another process - it succeeds. I doubt the drop table's nature is relevant here. Something wrong happens with the driver at some point and I cannot get what. I checked the driver's memory - no problem.
Any ideas on this puzzler?
UPDATE: I raised spark.executor.heartbeatInterval and both mentioned issues went away. Now I clearly see the issue: driver fails with Java Heap Space OOM. But what can be the reason for this if I am NOT pulling anything from the driver? I see a lot of broadcast/accumulator operations in logs, but they come from the engine, not from me.
UPDATE 2.0: Actually the idea of this question is quite simple: how sparkSQL query can lead to OOM in driver? What else can lead to OOM on driver? Logging policy?