Python read JSON files from all sub-directories

Viewed 3038

I have a following folder structure:

Directory    
    - Subdirectory 1:
       file.json
    - Subdirectory 2:
       file.json
    - Subdirectory 3:
       file.json
    - Subdirectory 4:
       file.json

How do I read these JSON files using Pandas?

2 Answers

You could do the following:

import glob, os
working_directory = os.getcwd()

sub_directories = [active_directory + "/" + x for x in os.listdir(working_directory) if os.path.isdir(active_directory + "/"+x)]
all_json_files = []

for sub_dir in sub_directories:
    os.chdir(sub_dir)
    for file in glob.glob("*.json"):
        all_json_files.append(sub_dir + "/" + file)

#Get back to original working directory
os.chdir(working_directory)

list_of_dfs = [pd.read_json(x) for x in all_json_files]

From there, if all json files have the same structure, you could concatenate them to get one single dataframe:

final_df = pd.concat(list_of_dfs)
Related