How to keep partition columns when reading in ORC files in Spark

user_3818487 0
Viewed 5263

When reading in an ORC file in Spark, if you specify the partition column in the path, that column will not be included in the dataset. For example, if we have

val dfWithColumn = spark.read.orc("/some/path") 

val dfWithoutColumn = spark.read.orc("/some/path/region_partition=1")

then dfWithColumn will have a region_partition column, but dfWithoutColumn will not. How can I specify that I want to include all columns, even if they're partitioned?

I am using spark 2.2 on scala.

EDIT: This is a re-usable Spark program that will take in its arguments from the command line; I want the program to work even if the user passes in a specific partition of a table instead of the whole table. So, using Dataset.filter is not an option.

orc apache-spark-sql apache-spark
3 Answers
user_65994940

Instead of adding your partitioned columns in the path, add them as filters. Modify your code to - 

val dfWithColumn = spark.read.orc("/some/path/").where($"region_partition" === 1)

This will identify the schema properly and will read data only for "region_partition=1" directory.

user_24996250

If the aim is to load one partition not the whole data, then you can get benefit of the lazy loading of spark and do the following:

val dfWithColumn = spark.read.orc("/some/path") 
dfWithColumn= dfWithColumn.where($"region_partition" === 1)

By doing this you will be getting data from the folder:

"/some/path/region_partition=1"

The win of this is that you keep the original structure with having the partition column inside your dataset.

But if you aim to manipulate the read dataset to add some column with some value I suggest to use the method:

withColumn

user_43854530
Related