Is there any way to read batch of events from event hub in databricks?

user_16034298 0
Viewed 605

My Requirement is i should read 1000 messages from event hub at a time in azure databricks and do some processing and then read the next 1000 messages in the queue.

On a whole i should continously read 1000 messages at a time.

azure-eventhub azure-databricks databricks apache-spark
1 Answers

You need to set value of maxEventsPerTrigger to 1000.

maxEventsPerTrigger - Rate limit on maximum number of events processed per trigger interval. The specified total number of events will be proportionally split across partitions of different volume.

Sample Code -

import org.apache.spark.eventhubs.

val cs = "<your-connection-string>"
val eventHubsConf = EventHubsConf(cs)
  .setConsumerGroup("sample-cg")
  .setMaxEventsPerTrigger(10000)

Refer - https://docs.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/streaming-event-hubs#eventhubsconf

user_111048050
Related