Read Excel files from S3 using Scala, Spark and org.apache.poi

Viewed 190

I'm looking for the way to open and process an Excel file (*.xlsx) in Spark job. I'm quite new to Scala/Spark stack so trying to complete it in pythonic way :)

Without Spark it's simple:

val f = new File("src/worksheets.xlsx")
val workbook = WorkbookFactory.create(f)
val sheet = workbook.getSheetAt(0)

But Spark needs some streaming input. I've configured Hadoop for S3 (in my case - MinIO)

val hadoopConf = sparkSession.sparkContext.hadoopConfiguration
hadoopConf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
hadoopConf.set(
      "fs.s3a.aws.credentials.provider",
      "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
)
hadoopConf.set("fs.s3a.path.style.access", "true")
hadoopConf.set("fs.s3a.access.key", params.minioAccessKey.get)
hadoopConf.set("fs.s3a.secret.key", params.minioSecretKey.get)
hadoopConf.set(
      "fs.s3a.connection.ssl.enabled",
      params.minioSSL.get.toString
)
hadoopConf.set("fs.s3a.endpoint", params.minioUrl.get)

val FilterDF = sparkSession.read
    .format("com.crealytics.spark.excel")
    .option("recursiveFileLookup", "true")
    .option("modifiedBefore", "2020-07-01T05:30:00")
    .option("modifiedAfter", "2020-06-01T05:30:00")
    .option("header", "true")
    .load("s3a://first/");
println(FilterDF)

So the question is: how to configure DataFrame (or, maybe some other solution) to filter and gather files in some time range from S3 bucket and make it suitable to work with Apache POI? Its Workbook can process general file objects as well as InputStream (so this might be the point of conversion)

Thanks in advance

0 Answers
Related