What would be the best way to read data parquet for data range using spark(scala/java) data frame?

Viewed 76

What would be the best way to read data parquet for data range using spark(scala/java) data frame?

The data is partitioned based on the date columns like year, month and day, wildcards can be used and what would be best way?

val dataframe = sqlContext
  .read
  .parquet("file:///your/path/data=jDD/year=2015/month=10/day={5,6}/*")

Wildcards can also be used to specify a range of months,days:

val dataframe = sqlContext
  .read
  .parquet("file:///your/path/data=jDD/year=2015/month=[1-5]/day=[5-10]/*")
0 Answers
Related