Takes forever to read and flatten 100K+ json files

Viewed 38

I have large number of json file that i have to read, flatten and filter (select specific key-value). below is the code i am using

for i in range(len(json_files)):   # is the list of json files in directory
    text = []
    for index, js in enumerate(json_files):
        with open(os.path.join(path_to_json, js)) as json_file: # path_to_json is json directory
            jtext = json.load(json_file)
            json_text=jtext['KEY_TO_FILTER']
            text_json=flatten_data(json_text)  # function i'm using to select specific key in json
            text.append(text_json)

i really appreciate hint how to use multiprocessing. any other suggestion ow to speed up the process is very welcomed

0 Answers
Related