Loading JSON from URL Python Pandas

Question

I'm preparing learning materials for my students. For convenience I want to access the data from URL rather than asking them download it in advance. In this example I am trying to access bird drawings from Quick, Draw! google data set.

Here is working example from accessing data that's stored remotely with results commented out:

import pandas as pd
import os
import json
from glob import glob

# Convert top row to one dict
top_row_dict = lambda in_df: list(in_df.head(1).T.to_dict().values())[0]
# Load file from computer
base_dir = os.path.join('input', 'quickdraw_simplified')
obj_files = glob(os.path.join(base_dir, '*.ndjson'))
print(obj_files[0])
# input\quickdraw_simplified\full_simplified_bird.ndjson

c_json = pd.read_json(obj_files[0], lines = True, chunksize = 1)
# 

f_row = next(c_json)
# word  countrycode     timestamp   recognized  key_id  drawing
# 0     bird    US  2017-03-09 00:28:55.637750+00:00    True    4926006882205696    [[[0, 11, 23, 50, 72, 96, 97, 132, 158, 224, 2...

f_dict = top_row_dict(f_row)
# {'word': 'bird',
#  'countrycode': 'US',
#  'timestamp': Timestamp('2017-03-09 00:28:55.637750+0000', tz='UTC'),
#  'recognized': True,
#  'key_id': 4926006882205696,
#  'drawing': [[[0, 11, 23, 50, 72, 96, 97, 132, 158, 224, 255],
#    [22, 9, 2, 0, 26, 45, 71, 40, 27, 10, 9]]]}

However, when I try to do the same using API link, it fails:

import pandas as pd
import json

top_row_dict = lambda in_df: list(in_df.head(1).T.to_dict().values())[0]

url = 'https://console.cloud.google.com/storage/browser/quickdraw_dataset/full/simplified/bird.ndjson'
# Load dataset
c_json = pd.read_json(url, lines = True, chunksize = 1)
# 
f_row = next(c_json)
# __
f_dict = top_row_dict(f_row)
# IndexError: list index out of range

user_328036 · Answer

The URL you're trying to use requires login (since it links to the Cloud Console).

However, the dataset is stored in a Google Cloud Storage bucket that is publicly accessible.

This means that you can use the http://pypi.org/p/google-cloud-storage package to load the files directly from the bucket instead.

Something like:

from google.cloud import storage
client = storage.Client()
bucket = client.get_bucket('quickdraw_dataset')
blob = bucket.get_blob('full/simplified/bird.ndjson')

c_json = pd.read_json(blob, lines = True, chunksize = 1)
...

Loading JSON from URL Python Pandas

1 Answers