I have built a streaming pipeline with spark autoloader.
Source Folder is an azure blob container. We encountered a rare issue (could not replicate it). Below is the exception Message:
org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename temp file dbfs:/mnt/delta_checkpoints/sources/0/rocksdb/__tmp_path_dir/.2.zip.52d0723f-b803-4a8a-9533-9d6e67813641.tmp to dbfs:/mnt/delta_checkpoints/sources/0/rocksdb/2.zip because file exists
Please help on this with a resolution, as this looks like some known platform issue.