This may have been answered elsewhere but I couldn't find an exact solution I was looking for.
I have a dataframe df created using:
df=spark.sql('''select distinct Col_1, Col_2, Col_3 from sourceTable''')
with the following structure
| col_1 | Col_2 | Col_3 |
|---|---|---|
| first_row | row | line |
The target table has the following structure where Col_3, rundate and runtime are partition columns in hive
| col_1 | Col_2 | Col_3 | rundate | runtime |
|---|---|---|---|---|
| first_row | row | line |
Now normally when schema matches, I know I can write the df to hive using
df.write.mode("overwrite").partitionBy("Col_3").insertInto("TargetTable", overwrite=True)
and set
hive.exec.dynamic.partition = true
hive.ecec.dynamic.partition.mode = nonstrict
spark.sql.sources.partitionOverwriteMode = dynamic
The challenge I am facing is that my source schema is different and moreover, I want to partition by Col_3 and month (instead of daily rundate and runtime).
Can someone help me with this?