Is there any way to get the number of records written when using spark to save records? While I know it isn't in the spec currently, I'd like to be able to do something like:
val count = df.write.csv(path)
Alternatively, being able to do an inline count (preferably without just using a standard accumulator) of the results of a step would be (almost) as effective. i.e.:
dataset.countTo(count_var).filter({function}).countTo(filtered_count_var).collect()
Any ideas?
You could use an accumulator to count the rows as they are written out:
val count = df.sparkSession.sparkContext.longAccumulator("row count")
val counted = df.map(row => { count.add(1); row })(df.encoder)
counted.write.parquet("my file")
count.value
Since it has to decode/encode each row for us, I'm not sure this is faster than just checking the output:
df.sparkSession.read.parquet("my file").count
Parquet stores the row counts as metadata, so it might be fast enough to check.