Writing multiple partition specs to Apache Iceberg table

user_12289911 0
Viewed 606

I would like to write an Iceberg table with a different partition spec than the default table settings so that when I run data compaction the data would be compacted according to the default spec (as possible with the write-format config)

For example:

df.writeStream
  .format("iceberg")
  .outputMode("append")
  .options(Map("path" ->tableIdentifier,
                  "checkpointLocation" -> checkpointLocation,
                  "fanout-enabled" -> "true",
                   "spec-id" ->"3"))

Theoretically, it should be fine as Iceberg knows how to handle multiple partitions specs as mentioned here Partition Evolution. Practically, I haven't found a way to do it.

Is there any way to achieve this?

iceberg apache-spark-sql apache-spark
1 Answers

According to the docs Streaming writes, I think you can't specify partitions while writing stream. Maybe you can try adding the new partition field before writing, using ALTER TABLE SQL extensions available for Spark 3.x:

ALTER TABLE tmp.db.sample ADD PARTITION FIELD years(date)

For batch writes, partition expressions are now supported in partitionBy using Spark 3 new DataFrameWriterV2 API:

data.writeTo("tmp.db.table")
    .tableProperty("write.format.default", "orc")
    .partitionBy($"level", days($"ts"))
    .createOrReplace()
user_13865510
Related