Execute Data Factory pipeline based on event in table

Viewed 1151

I have a requirement to trigger Azure Data Factory pipeline whenever there is a new record in a table. Is there any way to achieve this.

5 Answers

No, Event-based trigger only support Azure Blob Storage by now. You can vote here to progress this feature in Azure Data Factory.

While this is not currently supported, here's an idea on how to fake it. (Just sharing an idea, not sure whether this is feasible.)

  1. Create a Trigger on INSERT
  2. Trigger executes a Stored Procedure
  3. Stored Procedure uses Polybase to create text file in Blob Storage with the relevant information (like new row ID).
  4. Create a BlobCreated event trigger over that Storage location in ADF or Logic App.

Doing this should end up with an Event Trigger that fires whenever a new row is inserted.

We can use logic app to trigger a pipeline based on a query that finds data inserted in past x minutes/seconds

There's no direct way to do this in ADF (yet). As others have pointed out, you can vote for the feature to be added and Microsoft might add it. But there's still a way to do this:

You can set up a Logic App. There's a really nice video tutorial on this here: https://www.youtube.com/watch?v=z0sMIN4xMSY

You can set up a logic app to use a SQL database as the trigger, and then you can decide if you want to have it trigger based on something new created in a specific table or something modified in a specific table. You can then action the logic app to trigger the run of an Azure data factory pipeline. You can set the Logic App to check the table as often as you like, for example, every minute, 3 minutes or longer etc.

If you want an ADF pipeline to run every time a certain table is modified or row inserted, but the table in question is very large, you can reduce compute by creating a track changes table. Basically have another, very small table that logs changes and then use this table for the one the logic app monitors. This would make it faster and more efficient in such a case.

Give the video a watch.

Related