I'm looking for the way to open and process an Excel file (*.xlsx) in Spark job. I'm quite new to Scala/Spark stack so trying to complete it in pythonic way :)
Without Spark it's simple:
val f = new File("src/worksheets.xlsx")
val workbook = WorkbookFactory.create(f)
val sheet = workbook.getSheetAt(0)
But Spark needs some streaming input. I've configured Hadoop for S3 (in my case - MinIO)
val hadoopConf = sparkSession.sparkContext.hadoopConfiguration
hadoopConf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
hadoopConf.set(
"fs.s3a.aws.credentials.provider",
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
)
hadoopConf.set("fs.s3a.path.style.access", "true")
hadoopConf.set("fs.s3a.access.key", params.minioAccessKey.get)
hadoopConf.set("fs.s3a.secret.key", params.minioSecretKey.get)
hadoopConf.set(
"fs.s3a.connection.ssl.enabled",
params.minioSSL.get.toString
)
hadoopConf.set("fs.s3a.endpoint", params.minioUrl.get)
val FilterDF = sparkSession.read
.format("com.crealytics.spark.excel")
.option("recursiveFileLookup", "true")
.option("modifiedBefore", "2020-07-01T05:30:00")
.option("modifiedAfter", "2020-06-01T05:30:00")
.option("header", "true")
.load("s3a://first/");
println(FilterDF)
So the question is: how to configure DataFrame (or, maybe some other solution) to filter and gather files in some time range from S3 bucket and make it suitable to work with Apache POI? Its Workbook can process general file objects as well as InputStream (so this might be the point of conversion)
Thanks in advance