Returning all Data from a CKAN API Request? (Python)

user_10890297 0
Viewed 172

This is my first time using a CKAN Data API. I am trying to download public road accident data from a government website. It is only showing the first 100 rows. On the CKAN documentation it says that the default limit of rows it requests is "100".I am pretty sure you can write an ckan expression to the end of the url to give you the max rows but I am now sure how to write it. Please see python code below of what I have so of far. Is it possible? Thanks

enter image description here

enter image description here

is there any way I can write code similar to the psuedo ckan code request below?

url='https://data.gov.au/data/api/3/action/datastore_search?resource_id=d54f7465-74b8-4fff-8653-37e724d0ebbb&limit=MAX_ROWS'

CKAN Documentation reference: http://docs.ckan.org/en/latest/maintaining/datastore.html

ckan pandas request python
1 Answers

There are several interesting fields in the documentation for ckanext.datastore.logic.action.datastore_search(), but the ones that pop out are limit and offset.

limit seems to have an absolute maximum of 32000 so depending on the amount of data you might still hit this limit.

offset seems to be the way to go. You keep calling the API with the offset increasing by a set amount until you have all the data. See the code below.

But, actually calling the API revealed something interesting. It generates a next URL which you can call, it automagically updates the offset based on the limit used (and maintaining the limit set on the initial call).
You can call this URL to get the next batch of results.

Some testing showed that it will go past the maximum though, so you need to check if the returned records are lower than the limit you use.

import requests


BASE_URL = "https://data.gov.au/data"
INITIAL_URL = "/api/3/action/datastore_search?resource_id=d54f7465-74b8-4fff-8653-37e724d0ebbb"
LIMIT = 10000


def get_all() -> list:
    result = []
    resp = requests.get(f"{BASE_URL}{INITIAL_URL}&limit={LIMIT}")
    js = resp.json()["result"]
    result.extend(js["records"])
    while "_links" in js and "next" in js["_links"]:
        resp = requests.get(BASE_URL + js["_links"]["next"])
        js = resp.json()["result"]
        result.extend(js["records"])
        print(js["_links"]["next"]) # just so you know it's actually doing stuff
        if len(js["records"]) < LIMIT:
            # if it returned less records than the limit, the end has been reached
            break
    return result

print(len(get_all()))

Note, when exploring an API, it helps to check what exactly is returned. I used the simple code below to check what was returned, which made exploring the API a lot easier. Also, reading the docs helps, like the one I linked above.

from pprint import pprint
pprint(requests.get(BASE_URL+INITIAL_URL+"&limit=1").json()["result"])
user_92672960
Related