I have a stream of data which I can write to an in-memory table with pyspark and query with sql from there:
transformeddata.writeStream\
.format("memory")\
.queryName("transformeddatatable")\
.trigger(processingTime='5 seconds')\
.start()\
.awaitTermination(20) # write the data for 20 seconds into the memory table from the stream
In the next cell I can query the data:
%%sql
SELECT * FROM transformeddatatable
This works well and the data appears in the in-memory table transformeddatatable.
However, I cannot manage to display the data directly in the console:
transformeddata.writeStream\
.format("console")\
.outputMode("append")\
.trigger(processingTime='5 seconds') \
.option("checkpointLocation", "tmp/checkpoint/streamtoconsole/")\
.start()\
.awaitTermination(20)
The only thing that gets returned to the console is a boolean value of False once it terminates after 20 seconds.
In a lot of examples online the code above works. I'm dumbfounded what I do wrong - is it a problem of Azure's Synapse Notebook? Does it only work with Databricks?