I have a complex Spark SQL query.
I want to use the History Server to identify the longest-running stages of the query to target that portion for refactoring.
My challenge now: if I look at the DAG for a particular stage, there is no indicator of what specific files or SQL operations resulted in the operations in the stage. It shows me things like "ZippedPartitionsRDD2 [190]", "MappedPartitionsRDD [191]" etc. with no context to what part of the query I'm in (or at least what
I can get a better sense of the query structure from the SQL view, but it's hard to pinpoint the long-running part in a bigger graph without some visual indicator.
Are there better ways to track down the part of the Spark query that are causing the long stages?