ADF Dataflow to connect destination on premise sql database to insert data in the table

Viewed 88

I am trying to fetch the data from storage account to on premise SQL server. in this regards, i am using ADF data flow to use sink destination system which is self hosted runtime to connect on premise SQL database. if I am validate my dataflow, getting below alert message. kindly help me.

My project architecture is (CRM dataverse) azure synapse link --> load the data (storage gen2) --> ADF (data flow)--> on Premise SQL validate alert message as below :

Linked service with Self-hosted Integration runtime is not supported in data flow. Please utilize the Azure IR with managed vnet

without self hosted runtime, we could not connect external network, please suggest, how do we resolve it.

2 Answers

Data flow now only support Azure IR so it doesn’t support on-premise dataset enter image description here

If your goal is to use data transformations in ADF using Mapping Data Flows with on-prem data, then build a pipeline with a Copy Activity first. Use the Self-Hosted Integration Runtime with the Copy Activity to stage your data in Blob Store. Then add a subsequent Execute Data Flow activity to transform that data.

video for reference:

https://www.youtube.com/watch?v=IN-4v0e7UIs

The Data flows only supports Azure Integration runtime. If your source data doesn't need any data flow transformations and if you want only a copy to on-prem SQL database, you can directly use copy activity which supports self-hosted IR.

If your source data from storage needs dataflow transformations, then first transform source data to storage account using Azure IR in data flows. After that, use copy activity to copy the transformed data from storage account to on-prem SQL database.

Please go through the below sample demonstration:

First create the required datasets. In this sample we need 3 datasets. one for source data from storage account, second one to store transformed data in storage account and the last one is on-prem dataset using SQL server linked service and self hosted IR (Here I am using Azure SQL database for sample).

Source data from storage account:

Used csv for sample.

enter image description here

Dataset to store transformed data:

enter image description here

For final target, give your on-prem dataset in copy activity sink.

Use Data flow to trasform:

Give the source and sink datasets.

enter image description here

In the sink settings give our same file name to transform the data to a single file in storage account.

enter image description here

Now the data will be stored in the above storage account. Use this as source in the copy activity.

Copy activity to copy transformed data to on-prem:

Source:

enter image description here

In sink give your on-prem dataset.

You can get the data like below.

enter image description here

NOTE: Here I have used Azure SQL database in copy sink, you can use on-prem SQL database .

Related