Azure Data Factory Pagination Rules - QueryParameters.page

user_15426519 0
Viewed 817

I'm syncing a REST API where we don't have a next page, but we do have the total pagecount as a viable option. This loops over the REST API 8 times just fine, but the problem is, it's not incrementing the page query parameter by 1 each time, which results in it fetching the first page 8 times.

I've tried these variations of the baseurl & relativeurl:

baseurl: https://api.****.com/****/export/****.json?lastModifiedStart=****&lastModifiedEnd=****&page=1
baseurl: https://api.****.com/****/export/****.json?lastModifiedStart=****&lastModifiedEnd=****

baseurl: https://api.****.com/****
relativeurl: export/****.json?lastModifiedStart=****&lastModifiedEnd=****&page=1
relativeurl: export/****.json?lastModifiedStart=****&lastModifiedEnd=****

with this as the pagination rule:

"paginationRules": {
"QueryParameters.page": "RANGE:1:$.metadata.pagecount:",
"supportRFC5988": "true" }

Here's what the $.metadata returns from the api:

"metadata": {
"path": "export/****.json",
"receivedat": "2021-12-21T13:01:57-0500",
"endrow": 10000,
"resourcename": "export",
"isprivate": 0,
"page": 1,
"startrow": 1,
"timezone": "America/New_York",
"x-format-date": false,
"pagecount": 8,
"recordcount": 74081 }

What am I doing wrong?

EDIT: Image Link of Setup

azure-data-factory-pipeline azure-data-factory-2 azure-data-factory rest pagination
1 Answers

I found out that the paginationRule for QueryParameters was turning page into essentially a variable. This was one of the page requests it made:

baseurl: ...?lastModifiedStart=2021-01-01&lastModifiedEnd=2021-12-31&7Size=1000&7=1

This tells me that page was being treated as a literal parameter for the url and was incrementing 'page' to be 1-8 instead of the value of page, so to fix this, I used the following setup:

baseurl: ...?lastModifiedStart=2021-01-01&lastModifiedEnd=2021-12-31&pageSize=1000&page=pagenum

and this as the pagination rule:

"paginationRules": {
"QueryParameters.pagenum": "RANGE:1:$.metadata.pagecount:",
"supportRFC5988": "true" }

After doing this, the requests worked like a charm!

user_154265190
Related