In order to speed up a merge operation in Databricks with PySpark, I ran the following commands:
ALTER TABLE delta.`/mnt/tbl` SET TBLPROPERTIES (delta.targetFileSize = 512000000);--set file size
OPTIMIZE delta.`/mnt/tbl`
In Databricks I could see that files were being shuffled, resized etc. However in the Azure storage container as well as DBFS this table is still made up of tens of thousands of files with various sizes, from 0B to 1GB. I can't find sources online explaining the exact result from this command - should the files in storage be grouped into sizes of 512MB? If not, then what actually gets grouped according to the targetFileSize? And should I find a way to resize the files of the table in storage? I assumed the table rows/cols/values are saved in those files in the containers in Azure storage, is this not the case?