Sort files by numerical order while excluding files with non-numeric filenames

user_2778860 0
Viewed 75

I have a bunch of files in a directory from which I would like to select the CSV files and sort them numerically because I will later apply some processing based on the numerical order. However, some of these files have non-numeric file names; I would like to ignore those and only sort and retrieve the files whose names are completely numerical (e.g. take 5.csv but ignore data5.csv, .DS_Store).

To give an example of my folder structure, my file names look like: 1.csv, 1.txt, 10.csv, 2.csv, 3.csv, .DS_Store. From those I want to yield the following sorted output: 1.csv, 2.csv, 3.csv, 10.csv and simply print out an error message for the other files that are not CSV files or have non-numeric names.

I tried to achieve this by adding a key to sorted function and using os.scandir(). Here is a minimal example.

import os

def get_csv_files(path):
    """
    Generator function to sort and yield the CSV files with numerical filenames.

    Arguments:
        path: Path to the directory containing the CSV files.
    Returns:
        Paths of the found CSV files.
    """
    for entry in sorted(os.scandir(path), key=lambda e: int(os.path.splitext(e.name)[0])):
        if entry.is_file() and entry.name.endswith(".csv"):
            yield entry.path
        elif entry.is_dir():
            yield from get_csv_files(entry.path)
        else:
            print(f"Neither a csv file, nor a dir: {entry.path}")

path = '/path/to/dataset'
for i, file in enumerate(get_csv_files(path)):
    print(file)

This works fine if the filenames only contain digits but fails otherwise with the following error (for the filenames given above):

ERROR with .DS_Store
  File "test.py", line 12, in <lambda>
     for entry in sorted(os.scandir(path), key=lambda e: int(os.path.splitext(e.name)[0])):
 ValueError: invalid literal for int() with base 10: '.DS_Store'

I understand that the error is caused because my code tries to convert the string .DS_Store to int which is obviously wrong behaviour but I can't figure out how to do the sorting otherwise. I need a way to exclude the undesired filenames from the sorting key and probably need to manipulate the lambda function.

sorting python
2 Answers

Try this:

import os

path = "/absolute/path/to/files/to/sort/"
numerical_files = []
for f in os.listdir(path):
    file_name, extension = f.split('.')
    if extension == "csv" and file_name.isnumeric():
        numerical_files.append(file_name)
    else:
        print("Invalid file: ", f)

numerical_filenames_ints = [ int(f) for f in numerical_files ]
numerical_filenames_ints.sort()

for f in numerical_filenames_ints:
    file = str(f) + ".csv"
    print(file)
user_107653270

Add this function:

def listdir_nohidden(path):
    for f in os.listdir(path):
        if not f.startswith('.'):
            yield f

and change this line :

#for entry in sorted(os.scandir(path), key=lambda e: int(os.path.splitext(e.name)[0])):
for entry in sorted(listdir_nohidden(path), key=lambda e: int(os.path.splitext(e.name)[0])):
user_67780790
Related