My API requests only work if I limit my df to 1000 rows but not for full df consisting of 33000 rows

Viewed 95

So I've created a dataframe which contains only postcodes and I'm using an api that outputs longitude/latitude json info when I feed it the postcodes df. I've created a function for this converter and used panda apply on it. I can only print my dataframe if I drop everything but the first 1000 rows.


#code only works if I trim it down to first 1000
#geo_data.drop(geo_data.index[1000:33227],inplace=True)

def postcode_converter(x):
    longitude,latitude= None,None
    new_url='https://api.getthedata.com/postcode/'+str(x)
    new_url=new_url.strip()
    r = requests.get(new_url)
    time.sleep(0.1) 
    r= r.text
    try:
        result= json.loads(r)
        longitude=result['data']['longitude']
        latitude=result['data']['latitude']
    except Exception:
        pass
    return (longitude,latitude)

geo_data['longitude'],geo_data['latitude']=zip(*geo_data['postcode'].apply(postcode_converter))
print(geo_data) 
1 Answers

Seeing as it's a free API I think they have some form of rate limiting in place to stop abuse, hence it stopping after 1000 requests.

Looking at their website, you can download the same data that you are querying as a flat .csv file here: https://www.getthedata.com/open-postcode-geo

Reading that file into another dataframe you can simply do a merge on the postcode field to get your geolocation data.

Related