How to find the number of Inserts and Updates of Merge command?

user_1704348 0
Viewed 173

I have code similar to this in Spark(Scala). I would like to know the number of records this code updated/inserted when execute() is complete. Is there a way?

 deltaTable
     .as("target")
     .merge(
       source.as("source"),
       "target.key = source.key")
     .whenMatched
     .updateExpr(Map(
       "value" -> "source.value"))
     .whenNotMatched
     .insertExpr(Map(
       "key" -> "source.key",
       "value" -> "source.value"))
     .execute()
delta-lake apache-spark scala
1 Answers

You can get this data from the Delta table history - in Scala it's just

val lastOperationDF = deltaTable.history(1)

Then you get DataFrame, where you have the operationMetrics column that is a map from metric name to value. According to the documentation about metrics, information that you need is in the keys numTargetRowsInserted & numTargetRowsUpdated.

Alex_Ott_186270
Related