Spark overwrite removes privileges of already existing tables in db2

user_4938051 0
Viewed 803

I am using a spark cluster to move parquet files into DB2 tables. These DB2 tables are defined and pre-made, but empty. It is also granted some privileges to some users. However, if I use this code in spark:

ds.get
    .coalesce(1)
    .write
    .mode(SaveMode.Overwrite)
    .jdbc(fullJdbcUrl, tableName, props)

to move the parquet data, it removes the grant status (in SYSIBMADM) and seems to remove and create a table, instead of using what already exists. I don't want to use SaveMode.Append because there may be data in it already in some scenarios. Is there a way I can delete the rows in a table inside Spark (So that I can use Append afterwards), or for it to be forced into using the already-created table? Thank you.

apache-spark db2
1 Answers

In 2.1, Spark added support for truncate JDBC DataFrameWriter.

ds.get
    .coalesce(1)
    .write
    .mode(SaveMode.Overwrite)
    .option("truncate", true)
    .jdbc(fullJdbcUrl, tableName, props)

Here is the detailed information about truncate

user_13629260
Related