Why my delta lake table is not collecting statistics (min, max values)?

user_9885786 0
Viewed 756

Using delta lake on azure databricks 7.4.

I am trying to understand optimization, ZORDER and data skipping. I want to use ZORDER BY on the business date column : request_date_id (data type is integer). I understood that for implementing zorder, the targeted column should have delta statistics.

While my column request_date_id is in field number 7, it seems that my table does not have collected statistics on it

enter image description here

The table does have different distinct request_date_id. Why no statistics have been collected ?

enter image description here

It seems to me that data skipping options have been removed from databricks 7 onwards. But according to what i understood, it is still default. Delta lake table should still be collecting statistics.

Any suggestion or guidance would be very much appreciated. Many thanks in advance !

delta-lake azure-databricks databricks apache-spark indexing
1 Answers

What was removed in DBR 7.x is a special type of indexes implemented on the SQL level. But the Delta has built-in data skipping independent of if data registered as SQL table or just saved on disk. You can check that by looking into _delta_log/NNNNNN.json files, for each added file, there will be stats field, with min/max values for indexed columns. For example: {"add": {"path": "part-00002-d0b474c0-6b00-4bef-9fbb-de97e157e199-c000.snappy.parquet", "partitionValues": {}, "size": 134637, "modificationTime": 1628835148000, "dataChange": true, "stats": "{\"numRecords\":33333, \"minValues\":{\"id\":1}, \"maxValues\":{\"id\":99997}, \"nullCount\": {\"id\":0}}"}}

Alex_Ott_186270
Related