I am trying to insert data from a data frame into a Hive table. I have been able to do so successfully using
df.write.insertInto("db1.table1", overwrite = True).
I am just a little confused about the overwrite = True part -- I tried running it multiple times and it seemed to append, not overwrite. There wasn't too much in the docs, but when should I set overwrite to False vs. True?
df.insertInto works only if table already exists in hive.
df.write.insertInto("db.table1",overwrite=False) will append the data to the existing hive table.
df.write.insertInto("db.table1",overwrite=True) will overwrite the data in hive table.
Example:
df.show()
#+----+---+
#|name| id|
#+----+---+
#| a| 1|
#| b| 2|
#+----+---+
#save the table to hive
df.write.saveAsTable("default.table1")
#from hive
#hive> select * from table1;
#OK
#a 1
#b 2
df.write.insertInto("moch.table1",overwrite=True)
#from hive
#hive> select * from table1;
#OK
#a 1
#b 2
#appending data to hive
df.write.insertInto("moch.table1",overwrite=False)
#from hive
#hive> select * from table1;
#OK
#a 1
#b 2
#a 1
#b 2