Urban Institute API max size

user_10740427 0
Viewed 25

I'm trying to write a Python wrapper for accessing data from the Urban Institute datasets link to the docs but the output caps at 3000 rows. The docs don't mention anything about authentication or how to bypass this limit. Below my code:

url = "https://educationdata.urban.org/api/v1/schools/ccd/directory/2014/?fips=26"
response = urllib.request.urlopen(url)
data = pd.json_normalize(json.loads(response.read()), record_path=['results'])

This dataset caps at 3k, but if you paste directly to the browser, you can see that there are more than 3.5k.

api python json
1 Answers

The problem was that I needed to retrieve each one of the pages of the API response, below the code that pulls all the pages.

import requests
url = "https://educationdata.urban.org/api/v1/schools/ccd/directory/2014/?fips=26"
response = requests.get(url)
data = response.json()
while data['next'] is not None:
    response = requests.get(data['next'])

Happy API scrapping!

user_107404270
Related