Pyspark: Delta table as stream source, How to do it?

user_9562188 0
Viewed 3166

I am facing issue in readStream on delta table.

What is expected, reference from following link https://docs.databricks.com/delta/delta-streaming.html#delta-table-as-a-stream-source Ex:

spark.readStream.format("delta").table("events")  -- As expected, should work fine

Issue, I have tried the same in the following way:

df.write.format("delta").saveAsTable("deltatable")  -- Saved the Dataframe as a delta table

spark.readStream.format("delta").table("deltatable") -- Called readStream

error:

Traceback (most recent call last):
File "<input>", line 1, in <module>
AttributeError: 'DataStreamReader' object has no attribute 'table'

Note: I am running it in localhost, using pycharm IDE, Installed latest version of pyspark, spark version = 2.4.5, Scala version 2.11.12

delta-lake pyspark databricks apache-spark
2 Answers

The DataStreamReader.table and DataStreamWriter.table methods are not in Apache Spark yet. Currently you need to use Databricks Notebook in order to call them.

user_10388260

Try now with Delta Lake 0.7.0 release which provides support for registering your tables with the Hive metastore. As mentioned in a comment, most of the Delta Lake examples used a folder path, because metastore support wasn't integrated before this.

Also note, it's best for the Open Source version of Delta Lake to follow the docs at https://docs.delta.io/latest/index.html

user_73796440
Related