I am using apace spark structured streaming to stream the data from the delta table. The problem is that I have no visibility on consumer lag. How can we calculate consumer lag?
I have also gone throw the checkpoint files created per batch but that only has a timestamp in the epoch.
.format("delta")
.load("/mnt/delta/events")
.groupBy("customerId")
.count()
.writeStream
.format("delta")
.outputMode("complete")
.option("checkpointLocation", "/mnt/delta/eventsByCustomer/_checkpoints/streaming-agg")
.start("/mnt/delta/eventsByCustomer")