Spark read csv - multiple S3 paths in Java

Viewed 104

I am trying to read multiple s3 directories (each directory will have multiple files in it) using spark's read csv method however I get error as s3 path has some illegal character's. I have already checked related questions but don't see solution for java. Not able to implement the same solution for java.

DataSet<Row> DocsTemp  = null;
String scanResultFolder = "\"" + "s3a://somebucket/Dir1/" + "\",\"" + "s3a://somebucket/Dir2/" + "\"";
DocsTemp = spark.read().csv(scanResultFolder);

but when running, it considers entire string (scanResultFolder) as a single path and gives error.

Please suggest me the correct way to achieve this functionality.

1 Answers
Related