Is re-clustering of a z-ordering optimization a manual process in Databricks or is there a process running in the background that asynchronously reorders the cluster index after data has been inserted or updated?
1 Answers
It is not an asynchronous or continuous process today, but it is an idempotent operation. From the docs:
Prior to Databricks Runtime 5.0, Z-Ordering was not an idempotent operation, simply rewriting all data matching the given filter every time the command was run. Starting with Databricks Runtime 5.0, Z-Ordering is idempotent too, just like bin-packing.
You'll have to Z-order new rows in your table, but you won't have to rewrite the entire table to do so.
Related