I'm looking at the Spark UI (Spark v1.6.0) for a stage of a job I'm currently running and I don't understand how to interpret what its telling me:
The number of records in the "Shuffle Write Size / Records" column makes sense, those numbers are consistent with the data I'm processing.
What I do not understand is the numbers in "Input Size / Records". They indicate that the incoming data has only ~67 records in each partition; the job has 200 partitions so ~1200 records in all. I dont know what that is referring to, none of the input datasets to this job (which was implemented using SparkSQL) have ~1200 records in them.
So, I'm flummoxed as to what those numbers are referring to. Can anyone enlighten me?