Processing an Event Stream in Azure Databricks

Viewed 40

I am looking to implement a solution to populate some tables in Azure SQL based on events that are flowing through Azure Event Hubs into Azure Data Lake Service (Gen2) using Data Capture.

The current ingestion architecture is attached:

Current Architecture

I need to find an efficient way of processing each event that lands in the ADLS and writing it into a SQL database whilst joining it with other tables in the same database using Azure Databricks. The flow in Databricks should look like this:

  1. Read event from ADLS
  2. Validate schema of event
  3. Load event data into Azure SQL table (Table 1)
  4. Join certain elements of Table 1 with other tables in the same database
  5. Load joined data into a new table (Table 2)
  6. Repeat steps 1-5 for each incoming event

Does anyone have a reference implementation that has delivered against a similar requirement? I have looked at using Azure data Factory to pick up and trigger a Notebook whenever an event lands in ADLS (note, there is very low throughput of events (~1 every 10 seconds), however that solution will be too costly.

I am considering the following options:

  1. Using Stream Analytics to stream the data into SQL (however, the joining part is quite complex and requires multiple tables
  2. Streaming from the Event Hub into Databricks (however this solution would require a new Event Hub, and to my knowledge would not make use of the existing data capture architecture)
  3. Use Event Grid to trigger a Databricks Notebook for each Event that lands in ADLS (this could be the best solution, but I am not sure if it is feasible)

Any suggestions and working examples would be greatly appreciated.

0 Answers
Related