I have started a spark streaming job which streams data from kafka.I have assigned only 2 worker nodes with 15gb disk for testing.Within 2 hours the disk is full and the status of these nodes is showing as unhealthy on YARN Resource Manager web interface, and I have checked HDFS web interface which shows the Block Pool has used 95% of disk space. The problem is I am not storing any data on the nodes, just reading from kafka, processing and storing to MongoDB.
1 Answers
The Dataproc base image takes at least a few GB of space, so you're left with let's say 10GB per worker.
There's two main usages of disk space I can think of:
1) If you've enabled checkpointing (e.g. ssc.checkpoint(dir)): https://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing. That's probably on HDFS.
If you think HDFS is the issue, you can ssh into the master node, and run hdfs dfs -ls -R / to find which files are taking up space.
2) Ephemeral shuffle data gets written to disk between stages
This is less likely in a streaming job, but it's worth checking if HDFS isn't using much space. You can run du to find the directory taking up space, and I bet it's in nm-local-dirs: https://linuxhint.com/disk_space_directory_command_line/
All of that being said, 15GB is a really, really small disk size. PD is relatively cheap compared to compute, and I would suggest just using a larger boot disk size. If you're looking to cut down on cost, consider using e2 machine types.
Related