How to filter the output of web activity of Rest API using Filter Activity in Azure Data Factory?

user_15521809 0
Viewed 265

I am using Azure Data Factory to create the Data pipelines and store the output as JSON files in the Azure blob storage account.

The issue is I am using the Web activity with rest API for retrieving the data and parsing the data along with authorization token to copy activity.

Now I want to filter the output of web activity by removing or skipping the data from web activity before parsing it to the copy activity. So I can save the data by removing or skipping data in the JSON files as I want.

Is it possible to filter the output of web activity before parsing to the copy activity?

azure-data-factory-pipeline azure-data-factory-2 azure-data-factory filter api
1 Answers

Here is a simple demo for your usecase. First , im using a jsonPlaceHolder data check this API:

https://jsonplaceholder.typicode.com/users

second , i tried to do the following in Azure Data Factory:

  1. Created a webActivity to call the api(Its a simple GET request - please check the response).
  2. created a set variable activity to save and parse data to be used later.
  3. filtered data according to my condition

Simple ADF pipeline

WebActivity :

Under settings, added the jsonPlaceHolder url and GET method, in headers : Content-Type : application/json.

SetVariable:

Under Variables , Name : data , Value : @json(activity('TestWebActivity').output.Response)

Filter Activity:

Under Settings , items : @variables('data') , condition : @startswith(item().username,'B')

Explanation:

here i set the variable name in set activity to "data", so now in variables you can use data anywhere in your pipeline, in Filter activity , i set the data array in "items" and basically each json in the array is reffered as "item()" , so in order to filter the json array based on a value , in condition you can select the key : items().keyName ..

in order to save your data to Blob storage , add a Copy activity after the filter and link the blob storage as a sink..

user_67888850
Related