Python OSError: [Errno 9] Bad file descriptor after opening big json file

Viewed 1175

I just tried to read in a big json file (the Wikipedia json dump) in Python line by line and got the Error:

Traceback (most recent call last):
  File "C:/.../test_json_wiki_file.py", line 19, in <module>
    test_fct()
  File "C:/.../test_json_wiki_file.py", line 12, in test_fct
    for line in f:
OSError: [Errno 9] Bad file descriptor

Here is my code:

import json

def test_fct():
    data = []
    i = 0
    with open('E:/.../20200713.json/20200713.json') as f:
        for line in f:
            data.append(json.loads(line))
            i = i + 1

        if i > 1:
            input_file.close()
            return data

test_data = test_fct()

The file size is around 700GB and the description (https://www.wikidata.org/wiki/Wikidata:Database_download) of the file states that it can be read line by line. I don't know if this is important but the E:/ hard drive is an external one.

Thank you for your help in advance :)

1 Answers

I don't have any firsthand knowledge on opening large files in python, but did you mean to have the path as 20200713.json/20200713.json. Is the first one actually a directory that has a .json extension? I'd also suggest trying to first load a smaller sample of the file (opening might be hard, so maybe just use the more command in terminal?).

Related