I am trying to read a XML file with the code:
spark.read
.format("com.databricks.spark.xml")
.option("rowTag", meta.getRowTag)
.load(inputFile)
When I try to read big file (~45633 lines), I get
WARN YarnSchedulerBackend$YarnSchedulerEndpoint:66 - Requesting driver to remove executor 1 for reason Container marked as failed: container_e314_00000000 on host: xxxxxx555.name.host.com. Exit status: 143. Diagnostics: [2022-09-23 09:17:54.456]Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143. Killed by external signal
My spark config is:
--master yarn --num-executors 4 --executor-cores 12 --executor-memory 70G --driver-memory 20G
If I change the executor-memory to more than 70 e.g.
--master yarn --num-executors 4 --executor-cores 12 --executor-memory 80G --driver-memory 20G
I get:
ERROR TransportClient:233 - Failed to send RPC 8384934812145028858 to /XXX.X.XX.XX:00000: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
What could be the problem here?