I have large number of json file that i have to read, flatten and filter (select specific key-value). below is the code i am using
for i in range(len(json_files)): # is the list of json files in directory
text = []
for index, js in enumerate(json_files):
with open(os.path.join(path_to_json, js)) as json_file: # path_to_json is json directory
jtext = json.load(json_file)
json_text=jtext['KEY_TO_FILTER']
text_json=flatten_data(json_text) # function i'm using to select specific key in json
text.append(text_json)
i really appreciate hint how to use multiprocessing. any other suggestion ow to speed up the process is very welcomed