How to retrieve the continuationToken from Rest API while using Azure Data Factory?

user_15521809 0
Viewed 236

I'm trying to implement Azure Data Factory's Copy Activity to copy data from an API to Azure Blob Storage. I have set up the source & sinks correctly so that when I trigger the pipeline it pulls and loads the first load of data but I am struggling with pagination.

When I trigger the pipeline It loads the first page correctly and afterward, it doesn't return the next continuation token for fetching the data from API. If I use it until or for-each activity the pipeline copies the data of the same continuation token endless until timeout.

When I run the Rest API call in Postman it returns the data and also the next continuation token as well. The continuation Token will be like 0000xxxx-00000-xxx00-000000xx000000 and the next continuation token is like 0000xxyy-00000-xxx00-000000yy000000.

My goal is to retrieve the data from Rest API using a continuation token and retrieve the next continuation token so that I can retrieve the next page data until the continuation token is null and store it in Azure Blob Storage with Azure Data Factory Pipeline.

I am able to retrieve the access token from the rest API and only the first page data.

Is there any way to solve this issue please let me know?

azure-data-factory-pipeline azure-data-factory-2 azure-data-factory rest pagination
1 Answers

Assuming that your web activity gives you an array of elements to be processed, you can do the following :

Chain a For Each activity to iterate through the array.

Inside the ForEach activity, you can have items as something like (assuming array of items as outputArray) : @activity('webActivity').output.outputArray

Inside the ForEach activity, you can have a parameterized web activity to iterate through each item in the array. Each item can be accessed using the expression : @item()

Please check this link : https://imgur.com/a/LE1CHvp

user_67888850
Related