Azure Data Factory - REST API duplicates when using request_header pagination rule

user_3884042 0
Viewed 204

I'm trying to pull data from REST API using ADF's bulit-in connector. In the API response header I have 'Link' value to the next page:

enter image description here

But when using pagination rules for REST API connector like this: enter image description here

It imports the same pages over and over again and causes obvious duplicates and redundant API calls.

I tried different combinations of name/value rules:

  • Headers['Link']: Headers.Link (casues same as above)
  • Headers.Link: Headers['Link'] (casues same as above)
  • AbsoluteUrl: Headers['Link'] (causes 404)

Any Idea how to solve this?

Thanks!

azure-data-factory azure rest pagination
0 Answers
Related