How can I use Python requests to paginate api calls with offset parameter

user_1264698 0
Viewed 10702

I need to make repeated api calls as there is a limit on each call of 1000 records. There are about 20,000 total records of which I test them , keep a sample, then need request the next 1000. The offset parameter is available.

p = getpass.getpass()
url = ("https://example.api.com/api/1.0/devices/all/?offset={}&include_cols=asset_no,name,service_level,building&type=physical" 
r = requests.get(url, auth=HTTPBasicAuth('admin', p))
data = json.loads(r.text)
payload = data["Devices"]

offset value should increment by 1000 each api call, (ex. offset = 1000, offset = 2000, offset = 3000, etc) until all pages retrieved.

How could I create a function that makes paginated api calls using this offset parameter? I believe a generator is what is needed, but I'm unable to understand the examples I've found, with the offset parameter I need to use.

python-requests python-2.7 api python
2 Answers

I have to keep this very general since you did not give further details and mentioned no API vendor.

Pagination can be done using a simple while loop.

The basic workflow is that while you are getting a pagination token in your response, keep making subsequent requests. In pseudo code that might look like this:

Page = GetPageOfItems();
//process the data from the page, or add it to a larger array, etc.
while( Page->cursor )
    Page  = GetPageOfItems(Page->cursor);
    //process the data again
end

Ref: https://medium.com/square-corner-blog/tips-and-tricks-for-api-pagination-5cacc6f017da

The implementation also depends on API details, e.g. does the data header contain the current offset and/or a hasMore key, e.g.

p = getpass.getpass()
offset=0

while True:
    url = ("https://example.api.com/api/1.0/devices/all/?offset=" + offset + "&include_cols=asset_no,name,service_level,building&type=physical" 
    r = requests.get(url, auth=HTTPBasicAuth('admin', p))
    data = json.loads(r.text)
    # Process the payload or add it to a list
    offset = data['offset'] # offset +1?
    hasMore = data['has-more']
    if not hasMore:
        break
user_82919490

In a general sense, pagination with limit and offset is a little formula :

number_of_offsets = total_items / limit_per_page

The offset parameter controls the starting point within the collection of resource results.

If you have a collection of 15 items to be retrieved from a resource and you specify limit=5, you can retrieve the entire set of results in 3 successive requests by varying the offset value: offset=0, offset=5, and offset=10.

The number offsets become the number of pages. So in our example above it would be :

offset = 0 : page 1 - 5/15
offset = 5 : page 2 - 10/15
offset = 10: page 3 - 15/15
user_126870610
Related