I'm loading the spark dataframe to Hive using Hivewarehouseconnector. Hive tables are partitioned and orc formatted. Dataframe can contain multiple partition values. I need to load those partition values in single load and overwrite if that partition was already there.
df.select(columns).write\
.format(HiveWarehouseSession().HIVE_WAREHOUSE_CONNECTOR)\
.mode('overwrite')\
.option('inferSchema', 'true')\
.option('table', tablename)\
.option('partition', partition_column)\
.save()
Got the below error. I'm passing right partition column name.
Caused by: java.lang.IllegalArgumentException: Invalid partition spec: partition_column
Caused by: java.lang.NullPointerException: Conf non-local session path expected to be non-null
I'm using CDH-7.1.7-1.How to accomplish the load?