Azure Data Factory - REST API Call Pagination

Question

I'm making a call for data in Data Factory and struggling to call the url in the "next_page" item.

This is an example of what the first API call returns:

{
"items": [
    {
        "title_one": "TTL-55924",
        "id": "CPT-TTL-64577_TTL-55924",
        "title_id": "TTL-64577"
    },
    {
        "title_one": "TTL-69015",
        "id": "CPT-TTL-79755_TTL-69015",
        "title_id": "TTL-79755"
    }
],
"next_page": "http://api.com/api/info?offset=5000&key=XXXxxxXXXxxx"

}

I'm not sure which options to use in the Pagination Rules of my Copy activity.

Currently I'm trying the option "AbsoluteURL" with the value "$['next_page']" but this just returns an error.

user_15969107 · Answer

If your API response contains the next page URL property, then the “AbsoluteUrl“ pagination rule is the correct option to load the next page in the Azure data factory.

The supported values for pagination rules are mentioned in this MS document.

As mentioned in the example from the above document, Facebook Graph API returns the response as,

{
"data": [
    …
    …
],
"paging": {
    …
    …
    },
    "previous": "https://graph.facebook.com/me/albums?limit=25&before=NDMyNzQyODI3OTQw",
    "next": "https://graph.facebook.com/me/albums?limit=25&after=MTAxNTExOTQ1MjAwNzI5NDE="
}
}

Note: Pagination value of a JSON path expression starts with “$”.

Your pagination in REST copy activity looks like this:

In your API, the pagination should look like

user_13624415 · Answer

I have a similar issue. I am trying to copy shopify data from the rest API and was able to get the 1st page of data, however i cannot figure out how to set the pagination. THe pagination is coming in the header response like this:

I have it set to this but this returns an error:

ADF ERROR: Failure happened on 'Source' side. ErrorCode=RestSourceCallFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 404 indicates failure. Request URL: https://joyfolie.myshopify.com/admin/api/2021-07/%3Chttps://joyfolie.myshopify.com/admin/api/2021-07/products.json?limit=50&page_info=eyJsYXN0X2lkIjo0NjI0ODcyMjEwNDkwLCJsYXN0X3ZhbHVlIjoiKk5FVyogQmFlIFNraXJ0IGluIE9jaHJlIiwiZGlyZWN0aW9uIjoibmV4dCJ9%3E;%20rel=%22next%22 Response payload:{"errors":"Not Found"},Source=Microsoft.DataTransfer.ClientLibrary,' Source Pipeline Copy Shopify Products

THis is how the link looks in the response header

https://joyfolie.myshopify.com/admin/api/2022-01/products.json?limit=50&page_info=eyJsYXN0X2lkIjo0NjI0ODcyMjEwNDkwLCJsYXN0X3ZhbHVlIjoiKk5FVyogQmFlIFNraXJ0IGluIE9jaHJlIiwiZGlyZWN0aW9uIjoibmV4dCJ9; rel="next"

Azure Data Factory - REST API Call Pagination

2 Answers