Spark SQL - Recursive reading in folder

Question

I try to use HiveContext to take advantage of some windows functions in HiveQL in SparkSQL. But it cannot help me to read recursively data file in a folder (partition folder by year and month).

My folders:

data/outputOozie/22/year=2016 on driver

data/outputOozie/22/year=2016/month=10 on driver`

data/outputOozie/22/year=2016/month=9 on driver

data/outputOozie/22/year=2016/month=10/1 on driver

data/outputOozie/22/year=2016/month=10/2 on driver

data/outputOozie/22/year=2016/month=10/3 on driver

data/outputOozie/22/year=2016/month=9/1 on driver

data/outputOozie/22/year=2016/month=9/2 on driver

data/outputOozie/22/year=2016/month=9/3 on driver

Here is how I initiate my Hive Context:

val conf = new SparkConf().setAppName("Extraction process for ").setIfMissing("spark.master", "local[*]")
val sc = SparkContext.getOrCreate(conf)
sc.hadoopConfiguration.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")
sc.hadoopConfiguration.set("parquet.enable.summary-metadata", "false")
sc.hadoopConfiguration.set("mapreduce.input.fileinputformat.input.dir.recursive","true")
sc.hadoopConfiguration.set("hive.mapred.supports.subdirectories","true")
//val hiveContext = sqlContext.asInstanceOf[HiveContext] 
val hiveContext = sqlContext.asInstanceOf[HiveContext]
hiveContext.setConf("spark.sql.parquet.compression.codec", "snappy")
hiveContext.setConf("mapreduce.input.fileinputformat.input.dir.recursive","true")
hiveContext.setConf("mapred.input.dir.recursive","true") 
hiveContext.setConf("hive.mapred.supports.subdirectories","true")

Read file:

hiveContext.read.parquet(URLDecoder.decode(partitionLocation.get.toString, "UTF-8")) ==> Exception: file not found

But it's ok for SQL Context:

val sqlContext = new SQLContext(sc)
sqlContext.setConf("spark.sql.parquet.compression.codec", "snappy")
sqlContext.setConf("mapreduce.input.fileinputformat.input.dir.recursive","true")

Thanks for any suggestions!!!!

Spark SQL - Recursive reading in folder

0 Answers