Is there a way to read only the delta between 2 versions of deltaLake? specifically in Synapse
I have tried
%%pyspark
import delta
from pyspark.sql.functions import col, asc
df_delta = spark.read.format("delta") \
.option("readChangeFeed", "true") \
.option("startingVersion", 0) \
.option("endingVersion", 1) \
.load("/pathtoDeltaLake")
display(df_delta)
But the above seems to read more than just the delta between 2 versions. (only 1 row was added in version 1)
Right now, the Delta Change Feed is available only on Databricks (and only starting with specific runtime version), so you can't use it on Synapse. The options readChangeFeed, startingVersion & endingVersion are simply ignored by the open source Delta library that you're using on Synapse.
So you either execute this code on the Databricks and write as a separate table that could be read from Synapse, or you read each version into a separate dataframes (using .option("versionAsOf", version)) and the use some code to find diff (like in this answer).