Streaming data from Cosmos DB using Apache Spark 3

Question

Streaming from CosmosDB using Spark 2 connector can be achieved using Changefeed.

https://docs.microsoft.com/en-us/azure/cosmos-db/spark-connector#streaming-reads-from-cosmos-db

How do we do the same in Spark 3? I am using Cosmos DB Apache Spark 3 OLTP Connector for SQL API (beta).

https://docs.microsoft.com/en-us/azure/cosmos-db/create-sql-api-spark#query-cosmos-db

Alex_Ott_18627 · Answer

Right now there is no released version of CosmosDB connector that supports Spark 3. But you can compile connector yourself from the 3.0 branch - it works for me, including the changefeed

Update based on the comment: beta versions are available: https://docs.microsoft.com/en-us/azure/cosmos-db/sql-api-sdk-java-spark-v3

Streaming data from Cosmos DB using Apache Spark 3

1 Answers