Error While Writing into a BigQuery table from Dataproc - Spark

Question

I am trying to insert data into a Big Query table from Dataproc - Spark. I'm using the following code:

df.write \
  .format("bigquery") \
  .option("temporaryGcsBucket","bucket/temp") \
  .save("project.datasource.table1")

Error:

pyspark.sql.utils.IllegalArgumentException: 'SaveMode is set to ErrorIfExists and Table project.datasource.table1 already exists. Did you want to add data to the table by setting the SaveMode to Append? Example: df.write.format.options.mode(SaveMode.Append).save()'

I inserted ".mode(SaveMode.Append)", but It didn't work.

user_12135668 · Answer

That will do:

df.write \
  .format("bigquery") \
  .option("temporaryGcsBucket","bucket/temp") \
  .mode("append") \
  .save("gcp-bankier.sof.table1")

Error While Writing into a BigQuery table from Dataproc - Spark

1 Answers