I have noticed that whenever we issue an on demand export command trying to export large amount of data to an external table which points to blob storage , for a long time (e.g. 20-30 mins) nothing appears in the destination and then suddenly after that data starts being written to. Does it mean that all this time ADX spends in 'buffering' stuff ? Shouldn't the cluster start exporting data continuously as soon as we submit the export command? What is it really doing in that initial long gap when it's apparently not exporting.
1 Answers
The command writes to temp files on disk and uploads them to storage when they either reach their max size, or after all data has been written. The amount of time it takes for the 1st file to be uploaded can be related to the external table definition. For example, if the external table is partitioned and each partition is relatively small, it won't upload the files to storage until each file reaches its max size (or at the end of the command).
Related