Is there any way to write data from azure databricks to azure cosmos db GREMLIN API

SHIBASHISH TRIPATHY 0
Viewed 79

I am trying to write the vertices and edges to cosmos db gremlin api through Azure databricks but unfortunately I am facing error. I tried changing different versions of cluster and maven libraries still no use.

Libraries: Databricks configuration : 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)

Maven library installed : com.azure.cosmos.spark:azure-cosmos-spark_3-2_2-12:4.11.1

This is the document which I followed.

https://github.com/Azure/azure-cosmosdb-spark#using-databricks-notebooks

There might be some library conflict issue is happening because in document all older versions configuration are present. If any one came across this Kindly help?

cosmosDbConfig = {
  "Endpoint" : "https://xxxxxxxx.gremlin.documents.azure.com:443/",
  "Masterkey" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "Database" : "sample-database",
  "Collection" : "sample-graph",
  "Upsert" : "true"
}

cosmosDbFormat = "com.microsoft.azure.cosmosdb.spark"

(cosmosDbVertices.write.format(cosmosDbFormat).mode("append").options(**cosmosDbConfig).save()) ```

Error: 
Py4JJavaError: An error occurred while calling o1113.save.
: java.lang.ClassNotFoundException: 
Failed to find data source: com.microsoft.azure.cosmosdb.spark. Please find packages at
http://spark.apache.org/third-party-projects.html
       
    at org.apache.spark.sql.errors.QueryExecutionErrors$.failedToFindDataSourceError(QueryExecutionErrors.scala:557)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:758)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:808)
    at org.apache.spark.sql.DataFrameWriter.lookupV2Provider(DataFrameWriter.scala:983)
    at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:293)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:258)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
azure-cosmosdb-gremlinapi azure-cosmosdb azure-databricks apache-spark azure
1 Answers

I tried to reproduce same thing in my environment, and I got same error.

enter image description here

To resolve this error, try to install com.azure.cosmos.spark:azure-cosmos-spark_3-2_2-12:4.12.2 libraries and also follow below code.

Code:

cosEndpoint = "https://xxxxxx.dxx.azure.com:443/"
cosMasterkey = "xxxx"
cosDatabase = "xxxx"
cosContainer = "xxxx"

cfg1 = {
  "spark.cosmos.accountEndpoint" : cosEndpoint,
  "spark.cosmos.accountKey" : cosMasterkey,
  "spark.cosmos.database" : cosDatabase,
  "spark.cosmos.container" : cosContainer,
}

#Sample dataframe
cosmosDbVertices=spark.createDataFrame((("ss1", "cat", 2, True), ("cc1", "dog", 2, False)))\
  .toDF("id","name","age","isAlive")

# writing data into cosmosdb
sf=cosmosDbVertices.write.format("cosmos.oltp").options(**cfg1).mode("APPEND").save()

Output:

enter image description here

BhanunagasaiVamsi0
Related