Is there a way to speed up running .history() on a DeltaTable if the Operation name is given and I only want to find the most recent occurrence of the operation? For instance: Given a delta table dt I want to find the most recent "DELETE" in dt.history() and grab the value from operationMetrics. Right now it seems the only way to do this is grab the entire history and filter out all given "DELETE" operations, then return the top most row from that. This is extremely slow. Is there another way to go about this, or possibly manually step through the history without grabbing the entire history?
0 Answers
Related