Trying to connect Azure Databricks with Azure Event hubs to send some properties to event hub and read it through splunk

user_16271734 0
Viewed 81

I am looking to connect Azure data bricks to Event hub and read it through splunk . Initially I was able to send a test message and was able to receive the events in splunk(It was possible using scala as per https://docs.microsoft.com/en-us/azure/databricks/scenarios/databricks-stream-from-eventhubs --> Send tweets to event hubs). Now I am trying to implement the same using python using the reference from https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-python-get-started-send ---> Send Event . When I try to pass the object-parameters, it throws an error like unable to import from Azure Event hubs

Can anyone help me understand how can I connect Azure databricks with Azure Eventhubs and include sending object -parameters ?

PS : I have added the necessary libraries required to the cluster as below :

  1. com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.18
  2. azure-eventhubs-spark_2.11
  3. com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.22
  4. org.apache.spark:spark-avro_2.12:3.1.1

I have checked it with different versions of libraries too.

Can someone help me with the syntax part on how to pass the object parameters in the format of key value pairs ?

azure-eventhub azure-databricks azure
1 Answers

It's simple to connect to EventHubs from Azure Databricks - just follow official documentation, specifically the section Writing Data to Event Hubs (example is for batch write):

writeConnectionString = "SharedAccessSignatureFromAzurePortal"
ehWriteConf = {
  'eventhubs.connectionString' : 
     sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(writeConnectionString)
}

# Write body data from a DataFrame to EventHubs. 
# Events are distributed across partitions using round-robin model.
ds = df \
  .select("body") \
  .write \
  .format("eventhubs") \
  .options(**ehWriteConf) \
  .save()

You need to construct the body column somehow - by encoding your data as JSON using the to_json(struct("*")), or encoding data as Avro...

But you also have a problem in your cluster configuration - specifically these two libraries: azure-eventhubs-spark_2.11 and com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.22 - they are for Spark 2, but you use Spark 3. Uninstall them, and restart the cluster.

Alex_Ott_186270
Related