How to solve Error of offset mismatch in Azure Databricks Autoloader cloudfiles source?

user_14901869 0
Viewed 145

This happens when some files are deleted from the data source that Autoloader stream is reading from.

try:
    raw_df = spark.readStream.format("cloudFiles") \
            .option("cloudFiles.format","csv") \
            .option("cloudFiles.includeExistingFiles", "true") \
            .option("cloudFiles.allowOverwrites", "true") \
            .option("cloudFiles.schemaLocation", 
                opPath.outputPath +"/checkpoints/" + storageAccountInfo.adlsContainerName) \
            .option("delimiter","\t")\
            .load(source)

  except Exception as f:
    print(f)

Error Image

databricks-autoloader azure-databricks
1 Answers

I realized this was happening because some files were deleted from the blob storage which was the source of cloudfiles stream.

user_149018690
Related