I tried some simple MemoryStream samples in Azure databricks:
val input = MemoryStream[T]
...
input.addData(...)
input.toDS()
.writeStream
.queryName("events")
.outputMode("update") // or complete
.format("console") // or memory
.start()
Spark Structured Streaming MemoryStream + Row + Encoders issue
Is proper event-time sessionization possible with Spark Structured Streaming?
However, when I run them I get
Job aborted due to stage failure: ... java.io.InvalidClassException: org.apache.spark.sql.execution.streaming.MemoryStream; no valid constructor
Is there any work around do this? Memorystream seems quite useful to do some tests before working with real datasets. The above code seems to work fine in my local cluster via Spark-shell but ideally I don't want to go back and forth between local and databricks clusters to test things.