Python importing csv files within subfolders

user_15361369 0
Viewed 891

Is there a way of importing all the files within folder1? Each csv file is contained within a subfolder. Below is the file structure.

C:/downloads/folder1 > tree /F

C:.
│   tree
│
├───2020-06
│       test1.csv
│
├───2020-07
│       test2.csv
│
├───2020-08
│       test3.csv
│
├───2020-09
│       test4.csv

I'm aware of glob, below, to take all files within a folder. However can this be used for subfolders?

import glob
import pandas as pd

# Get a list of all the csv files
csv_files = glob.____('*.csv')

# List comprehension that loads of all the files
dfs = [pd.read_csv(____) for ____ in ____]

# List comprehension that looks at the shape of all DataFrames
print(____)
jupyter glob python
3 Answers

Use the recursive keyword argument of the glob.glob() method:

glob.glob('**\\*.csv', recursive=True)
user_135524700

You can use os.walk to find all sub_folder and get the required files

here's a code sample

import os
import pandas as pd

path = '<Insert Path>'
file_extension = '.csv'
csv_file_list = []
for root, dirs, files in os.walk(path):
    for name in files:
        if name.endswith(file_extension):
            file_path = os.path.join(root, name)
            csv_file_list.append(file_path)

dfs = [pd.read_csv(f) for f in csv_file_list]
user_52365750

I found this on Kite's website, check it out

path = "./directory/src_folder"

text_files = glob.glob(path + "/**/*.txt", recursive = True)

print(text_files)
OUTPUT
['./directory/src_folder/src_file.txt', './directory/src_folder/subdirectory/subdirectory_file.txt']
user_151759820
Related