Delta table as a stream source. How to calculate consumer lag

Viewed 162

I am using apace spark structured streaming to stream the data from the delta table. The problem is that I have no visibility on consumer lag. How can we calculate consumer lag?

I have also gone throw the checkpoint files created per batch but that only has a timestamp in the epoch.

https://docs.microsoft.com/en-us/azure/databricks/delta/delta-streaming#:~:text=for%20a%20stream-,Delta%20table%20as%20a%20stream%20source,and%20tables%20as%20a%20stream.

  .format("delta")
  .load("/mnt/delta/events")
  .groupBy("customerId")
  .count()
  .writeStream
  .format("delta")
  .outputMode("complete")
  .option("checkpointLocation", "/mnt/delta/eventsByCustomer/_checkpoints/streaming-agg")
  .start("/mnt/delta/eventsByCustomer")
0 Answers
Related