Data Coming from Event Hub is in Avro Format which in Binary/unreadable. Need to convert in readable format in a specific schema.
Below is the code which is tried:
%scala
// Configuration parameters for connecting to Event Hubs.
// Build connection string with the above information
val connectionString = ConnectionStringBuilder("our connection string).setEventHubName(EventHubName).build
val customEventhubParameters =
EventHubsConf(connectionString)
.setMaxEventsPerTrigger(1)
val incomingStream = spark.readStream.format("eventhubs").options(customEventhubParameters.toMap).load()
val messages = incomingStream.withColumn("Body", $"body".cast(StringType)).select("Body")
Need to read in user defined .avsc schema and store it in Dataframe for later processing