Reading first N lines of all files via Spark Scala

user_6933993 0
Viewed 128

If we exclude utilities and RDD nested zipping of filenames via zipWithIndex, what are the elegant options of for every file to be processed, to remove / skip the first N records? Where N > 1.

A few of the presented options here seem to work with first line.

apache-spark scala
1 Answers

Something like this will work at the cost of a shuffle

import org.apache.sql.functions._
import org.apache.sql.expressions.Window

val ROWS_TO_SKIP=5
val filtered_df = df
  .withColumn("__file", input_file_name())
  .withColumn("__idx", monotonicallyIncreasingId())
  .withColumn("__file_idx", row_number().over(
        Window.partitionBy('__file).orderBy('__idx))
  )
  .filter('__file_idx > ROWS_TO_SKIP)
  .drop("__file","__idx","__file_idx")
user_31696690
Related