Read json line by line using generator

user_7013263 0
Viewed 3116

I have a json file which is pretty big, and a function which reads the json file.

I need to keep the state between function calls (the next time when the function will be called I do not need to read the json file from the first line instead I need the function to pick up from where it had remained), so the first thing that came into my mind was the generator protocol.

This is my first attempt, but the function behaviour is not the desired one:


def first_task(file_name):
    """"
    Read file_name line by line
    """
    import json
    data = []
    with open(file_name) as f: 
        for line in f:
            try:
                data.append(json.loads(line))
            except ValueError:  # includes simplejson.decoder.JSONDecodeError
                print ('Decoding JSON has failed')
    yield data

What I would like is when first_task("test.json") it's called for the first time to return the first line from the json, then when the function is called second time to return the second line from json, and so on until the EOF is reached.

Sample from json file:


{"venue":{"venue_name":"Datong High School","lon":0,"lat":0,"venue_id":23779799},"visibility":"public","response":"no","guests":0,"member":{"member_id":120119272,"photo":"http:\/\/photos3.meetupstatic.com\/photos\/member\/b\/2\/b\/c\/thumb_262125756.jpeg","member_name":"Allen Wang"},"rsvp_id":1658733801,"mtime":1489925470960,"event":{"event_name":"Play Intermediate Volleyball","event_id":"jkpwmlywgbmb","time":1491613200000,"event_url":"https:\/\/www.meetup.com\/Taipei-Sports-and-Social-Club\/events\/236786445\/"},"group":{"group_topics":[{"urlkey":"fitness","topic_name":"Fitness"},{"urlkey":"mountain-biking","topic_name":"Mountain Biking"},{"urlkey":"sports","topic_name":"Sports and Recreation"},{"urlkey":"outdoors","topic_name":"Outdoors"},{"urlkey":"fun-times","topic_name":"Fun Times"},{"urlkey":"winter-and-summer-sports","topic_name":"Winter and Summer Sports"},{"urlkey":"adventure","topic_name":"Adventure"},{"urlkey":"water-sports","topic_name":"Water Sports"},{"urlkey":"sports-and-socials","topic_name":"Sports and Socials"},{"urlkey":"hiking","topic_name":"Hiking"},{"urlkey":"excercise","topic_name":"Exercise"},{"urlkey":"recreational-sports","topic_name":"Recreational Sports"}],"group_city":"Taipei","group_country":"tw","group_id":16585312,"group_name":"Taipei Sports and Social Club","group_lon":121.45,"group_urlname":"Taipei-Sports-and-Social-Club","group_lat":25.02}}
{"venue":{"venue_name":"Cafe Vitus","lon":121.54731,"lat":25.052959,"venue_id":19712922},"visibility":"public","response":"no","guests":0,"member":{"member_id":221379606,"photo":"http:\/\/photos2.meetupstatic.com\/photos\/member\/8\/3\/c\/4\/thumb_263973732.jpeg","member_name":"Benita  Syu"},"rsvp_id":1658877353,"mtime":1489925471668,"event":{"event_name":"New Place! Every Saturday night multilingual café","event_id":"hvkmsmywfbhc","time":1490439600000,"event_url":"https:\/\/www.meetup.com\/polyglottw\/events\/238185973\/"},"group":{"group_topics":[{"urlkey":"language","topic_name":"Language & Culture"},{"urlkey":"language-exchange","topic_name":"Language Exchange"},{"urlkey":"chinese-language","topic_name":"Chinese Language"}],"group_city":"Taipei","group_country":"tw","group_id":18743595,"group_name":"Multilingual Cafe Language Exchange","group_lon":121.45,"group_urlname":"polyglottw","group_lat":25.02}}
{"venue":{"venue_name":"Panera Bread","lon":0,"lat":0,"venue_id":24945082},"visibility":"public","response":"yes","guests":0,"member":{"member_id":44748032,"photo":"http:\/\/photos4.meetupstatic.com\/photos\/member\/d\/7\/1\/6\/thumb_64255062.jpeg","member_name":"Valerie"},"rsvp_id":1658877355,"mtime":1489925472035,"event":{"event_name":"Meet & Greet Icebreaker conversations in Morris County","event_id":"236222256","time":1490389200000,"event_url":"https:\/\/www.meetup.com\/Mingle-Around-In-North-Jersey-Single-Events-Adventures\/events\/236222256\/"},"group":{"group_topics":[{"urlkey":"wine","topic_name":"Wine"},{"urlkey":"hiking","topic_name":"Hiking"},{"urlkey":"diningout","topic_name":"Dining Out"},{"urlkey":"marketing","topic_name":"Marketing"},{"urlkey":"newintown","topic_name":"New In Town"},{"urlkey":"socialnetwork","topic_name":"Social Networking"},{"urlkey":"women","topic_name":"Women's Social"},{"urlkey":"outdoors","topic_name":"Outdoors"},{"urlkey":"professional-networking","topic_name":"Professional Networking"},{"urlkey":"adventure","topic_name":"Adventure"},{"urlkey":"singles-30s-50s","topic_name":"Singles 30's-50's"},{"urlkey":"small-business-marketing-strategy","topic_name":"Small Business Marketing Strategy"},{"urlkey":"professional-singles","topic_name":"Single Professionals"},{"urlkey":"dating-and-relationships","topic_name":"Dating and Relationships"},{"urlkey":"singles-40s-50s","topic_name":"Singles 40's - 50's"}],"group_city":"Hackensack","group_country":"us","group_id":17370312,"group_name":"Mingle Around 30s 40s 50s (Single Events & Adventures)","group_lon":-74.05,"group_urlname":"Mingle-Around-In-North-Jersey-Single-Events-Adventures","group_state":"NJ","group_lat":40.89}}
generator python json
2 Answers

This should work. With no error checking.

def gen(file_name):
    with open(file_name) as fh:
        line = fh.readline()
        while line:
            yield json.loads(line)
            line = fh.readline()

In python 3.8 you could do it like this:

def gen(file_name):
    with open(file_name) as fh:
        while line := fh.readline():
            yield json.loads(line)
user_97880430

Although it is not quite clear on what you need, if you just need individual lines, instead of constructing a list you can just yield each line and then call the generator with next as you need it, since this will only the read the file once and supply individual lines as needed until it is exhausted:

## contents of example.txt

{"first": "line"}
{"second": "line"}
{"third": "line"}

Construction of line supplier:

def supply_line(file_name):
    with open(file_name) as fh:
        for line in fh:
            yield json.loads(line)

When you call it with next ensure to provide a default, so you don't have wrap around a try/except to catch a StopIteration exception:

producer = supply_line('file_with_json_lines.txt')

In [7]: next(producer, '')
Out[7]: {'first': 'line'}

In [8]: next(producer, '')
Out[8]: {'second': 'line'}

In [9]: next(producer, '')
Out[9]: {'third': 'line'}

## when the file is done, it will produce a default, which in this case is an empty string
In [10]: next(producer, '')
Out[10]: ''

In [11]: next(producer, '')
Out[11]: ''

If you need to start over, you can call the generator again, if you want to store all the lines in a list, you can cast the generator to list (however this will load the file everytime):

In [13]: all_lines = list(supply_line('file_with_json_lines.txt'))

In [14]: all_lines
Out[14]: [{'first': 'line'}, {'second': 'line'}, {'third': 'line'}]

And obviously a for loop:

In [15]: for line in supply_line('file_with_json_lines.txt'):
    ...:     print(line)
    ...:
{'first': 'line'}
{'second': 'line'}
{'third': 'line'}
user_621380
Related