Can I read only wanted Files from a directory in scala-spark

user_12254220 0
Viewed 564

I want to read some specific files whose names I have in an array. Can I read and store values in only those files. I an new to scala and spark, please help me out.

Entire files

val status = fs.listStatus(new Path("..../sample/t1988")) 

status.foreach(x=> println(x.getPath))

file:/......../sample/t1988/01001099999.csv
file:/......../sample/t1988/01001399999.csv
file:/......../sample/t1988/01001499999.csv

and so on

Names of file i want to read are in an array

arr: Array[Any] = Array(42044099999, 42045099999, 42060099999, 42063099999, 42075099999, 42077099999, 42079099999, 42080199999

How to read content of only the above files

apache-spark-sql scala
1 Answers

If you have an array files with paths to your csv files, you can read them using Spark as follows:

val df = spark.read.format("csv").load(files:_*)
user_102985510
Related