I'm streaming from a Delta Table (source) to a Delta Table (target) in Databricks
%python
df = spark.readStream \
.format("delta") \
.load(path/to/source)
query = (df
.writeStream
.format("delta")
.option("mergeSchema", "true")
.outputMode("append")
.trigger(once=True) # Every 30 min
.option("checkpointLocation","{0}/{1}/".format(checkpointsPath,key))
.table(tableName)
)
But it seems that in a point of time, the job starts to process "less" data that it should be processing:
Do you know if there is a max size to process streaming data or something?
I'm trying to debug reading the logs but i can't find any issue
