Issue Streaming Delta Table in Databricks

Viewed 323

I'm streaming from a Delta Table (source) to a Delta Table (target) in Databricks

%python

df = spark.readStream \
          .format("delta") \
          .load(path/to/source) 



query = (df
                .writeStream
                .format("delta")
                .option("mergeSchema", "true")
                .outputMode("append")
                .trigger(once=True) # Every 30 min
                .option("checkpointLocation","{0}/{1}/".format(checkpointsPath,key))           
                .table(tableName)
          )

But it seems that in a point of time, the job starts to process "less" data that it should be processing:

enter image description here

Do you know if there is a max size to process streaming data or something?

I'm trying to debug reading the logs but i can't find any issue

0 Answers
Related