How to get the latest insertion time for a delta table?

user_760807 0
Viewed 290

In my Spark structured streaming application, I have a code like this.

df = (
  spark.readStream.format("delta")
  .option("startingTimestamp", starting_time_stamp)
  .table(t)
)

Now if the given starting timestamp is later than the timestamp on which the last insertion was done, I get an error. So, my question is how can I check for the latest timestamp on which an insertion was committed?

spark-structured-streaming pyspark apache-spark
1 Answers

We can find the latest timestamp with the following code.

ts = spark.sql("SELECT max(timestamp) FROM (DESCRIBE HISTORY <table>)").first()[0]
user_7608070
Related