Create list of values from os.walk to show names of folders for each path in Python

Admin 1
Viewed 59

In my example directory structure, there are 3 folders under Project - das, des, and dga. There's also a file randomfile.xlsx Within each folder are files and subfolders.

Project
 ┣ das
 ┣ des
 ┃ ┣ subfolder_des1
 ┃ ┃ ┗ COM-EL-DES-0002-0.pdf
 ┃ ┗ COM-EL-DES-0001-0.pdf
 ┣ dga
 ┃ ┣ subfolder_dga1
 ┃ ┃ ┗ 3500-ST-DGA-0047-0.pdf
 ┃ ┣ subfolder_dga2
 ┃ ┃ ┣ 3500-ST-DGA-0046-0.pdf
 ┗ randomfile.xlsx

My code uses os.walk to return a directory tree list of values under the Project folder, which I add to a pandas DataFrame. e.g.

c:/Project/das
c:/Project/des
c:/Project/dga
c:/Project/des/subfolder_des1/
c:/Project/des//subfolder_des1/COM-EL-DES-0002-0.pdf
...
...

I'd like to create a second list which specifies the folder a value/path is associated with (e.g. das, des, and dga), but ignore subfolders. Any files in the Project folder would just have the value Project.

So resulting dataframe might look like:

Path                                                  col2
c:/Project/das                                         das
c:/Project/des                                         des
c:/Project/dga                                         dga
c:/Project/des/subfolder_des1/                         des
c:/Project/des/subfolder_des1/COM-EL-DES-0002-0.pdf    des
c:/Project/randomfile.txt                              Project
c:/Project/dga/subfolder_dga1/                         dga
...
...

Just unsure where to start, as I'm not sure I fully understand os.walk yet.

My code snippet:

def dir_tree(path):
    dir_list = []
    col2_list = []

    for root, dirs, files in os.walk(path):
        for d in dirs:
            dir_list.append(os.path.join(root, d))
        for f in files:
            dir_list.append(os.path.join(root, f))
    return dir_list, col2_list

def save_to_df(path):
    dir_list, col2_list = dir_tree(path)
    dir_tree_df = pd.DataFrame({"Path" : dir_list, "col2" : col2_list})

Edit: Something else to note, I'm only using dag, des etc as examples. I need to be able to run this script on any path, and it'll need to work out what the folder names are.

An updated script which might be doing what I want it to now:

def get_folder_name(path_str: str, source_directory: str) -> str:

    path_replace = path_str.replace(source_directory, "")
    path = Path(path_replace)
    parts = path.parts
    
    joined = os.path.join(source_directory, parts[1])
    if os.path.isfile(joined):
        return "root_files"
    else:
        return parts[1]
pandas os.walk python
1 Answers

You can use pathlib and it's .name and .parent attributes to create a function that attempts to find the desired folder name:

from pathlib import Path
import pandas as pd

def get_folder_name(path_str: str) -> str:
    """
    Given a path string, return the name of the desired folder name,
    either "das", "des", "dga", "Project", or None, when not any of the prior.

    Converts the path string to a Path object, and checks if the path.name is
    any of the folders that we want to check for. If not, we move the path
    to the parent instead, and try again.
    """
    path = Path(path_str)
    while path.name:
        if path.name in ("das", "des", "dga", "Project"):
            return path.name
        path = path.parent
    return None

# I've just copied these paths here for convenience, as obviously I don't
# have your file structure for testing. I've created a DataFrame like I'm
# sure you had previously.
paths = [
    "c:/Project/das",
    "c:/Project/des",
    "c:/Project/dga",
    "c:/Project/des/subfolder_des1/",
    "c:/Project/des/subfolder_des1/COM-EL-DES-0002-0.pdf",
    "c:/Project/randomfile.txt",
    "c:/Project/dga/subfolder_dga1/",
    "c:/Other/file.txt", # <- Custom, to show an edge case
]
df = pd.DataFrame({"Paths": paths})

# Here we `apply` get_folder_name on every element of the df["Paths"] column,
# and assign it to a new "Folder" column
df["Folder"] = df["Paths"].apply(get_folder_name)

print(df)

This program outputs:

                                               Paths   Folder
0                                     c:/Project/das      das
1                                     c:/Project/des      des
2                                     c:/Project/dga      dga
3                     c:/Project/des/subfolder_des1/      des
4  c:/Project/des/subfolder_des1/COM-EL-DES-0002-...      des
5                          c:/Project/randomfile.txt  Project
6                     c:/Project/dga/subfolder_dga1/      dga
7                                  c:/Other/file.txt     None

Edit as a response to the author's comment

If we can simply select the first folder after Projects, or alternatively just Projects if the path does not contain a folder under Projects, then the following snippet should work too, as a replacement of the get_folder_name function from above.


def get_folder_name(path_str: str) -> str:
    path = Path(path_str)
    parts = path.parts
    return parts[1] if len(parts) == 3 and path.is_file() else parts[2]

Here, we will first separate the path into its parts, e.g. path.parts ('c:\\', 'Project', 'des', 'subfolder_des1'). Then, we will usually pick parts[2], which is des in this example, with one very specific exception: If the parts tuple is only 3 long and the path itself points to a file, instead of a folder. In that case, we have e.g. c:\Projects\test.txt. Do note that this requires that Projects is at C:\Projects and not at C:\Bla\Projects etc., as that would require some modifications on the indices.

Note that this function isn't perfect, e.g. the path of C:\Projects fails, but this is mostly meant to guide you in the right direction. Feel free to modify this to your likings. Also, here's the pathlib documentation.

user_179363260
Related