WriteStream with format('console') in Azure Synapse Notebook

Viewed 119

I have a stream of data which I can write to an in-memory table with pyspark and query with sql from there:

transformeddata.writeStream\
   .format("memory")\
   .queryName("transformeddatatable")\
   .trigger(processingTime='5 seconds')\
   .start()\
   .awaitTermination(20)  # write the data for 20 seconds into the memory table from the stream

In the next cell I can query the data:

%%sql
SELECT * FROM transformeddatatable

This works well and the data appears in the in-memory table transformeddatatable.

However, I cannot manage to display the data directly in the console:

transformeddata.writeStream\
   .format("console")\
   .outputMode("append")\
   .trigger(processingTime='5 seconds') \
   .option("checkpointLocation", "tmp/checkpoint/streamtoconsole/")\
   .start()\
   .awaitTermination(20)

The only thing that gets returned to the console is a boolean value of False once it terminates after 20 seconds.

In a lot of examples online the code above works. I'm dumbfounded what I do wrong - is it a problem of Azure's Synapse Notebook? Does it only work with Databricks?

0 Answers
Related