Sorting directory by date Python

user_14608493 0
Viewed 341

I have the following directory format:

logs
---- abc_090000_20210124
---- abc_091500_20210124
etc...

The format is abc_(HHMMSS)_(YYYYMMDD)

I want to get the two most recent files by date and time.

If they are all on the same day I use this code:

directory = os.listdir("logs")
base = "abc"

date = sorted([x.split("_")[2] for x in directory])[-1]

t_of_day = sorted([x.split("_")[1] for x in directory if x.split("_")[2] == date])

a = f"logs/{base}_{t_of_day[-1]}_{date}"
b = f"logs/{base}_{t_of_day[-2]}_{date}"
print(a,b)

and it works properly, however, at the change of day the files will change like so:

...
---- abc_234506_20210124
---- abc_000007_20210125
...

and my code will give an error as there will not be two values in the list t_of_day.

I am unsure of how I could change this to suit my situation.

I have tried to use a condition where if date[-1] == date[-2] to use a different value, but as the dataset builds up this may become unreliable.

Is there perhaps an alternative way to sort the files? I must state that changing the file names is not an option.

python
4 Answers

First, parse the filenames as datetimes, then take the top k of those (could sort but it's faster to use heapq):

from datetime import datetime
import heapq

files = ...  # e.g.: use glob, listdir, whatever
last2 = heapq.nlargest(2, [
    (f, datetime.strptime('_'.join(f.split('_')[-2:]), '%H%M%S_%Y%m%d'))
    for f in files
], key=lambda ft: ft[1])

last2 contains tuples (filename, datetime). To get just the filenames (here on some synthetic list of filenames):

>>> [f for f, t in last2]
['abc_091500_20210124', 'abc_090000_20210124']
user_7581740

You can supply your own function as the key argument to determine how a list is sorted.

import os

def date_sort(filename):
    abc, hms, ymd = filename.split('_')
    return ymd + hms

filenames = os.listdir('logs')
filenames.sort(key=date_sort)

You could also use Python's datetime class, but I don't think it's necessary for simply sorting values like this. All you need are the units of time in descending size: year, month, day, hour, minute, second.

For just getting the two most recent, you don't really need to sort the whole list, but you can still use this same key function for "sorting" a heap queue. (Here's another answer that gives a good example using heapq.)

import os
import heapq

def date_sort(filename):
    abc, hms, ymd = filename.split('_')
    return ymd + hms

filenames = os.listdir('logs')
first, second = heapq.nlargest(2, filenames, key=date_sort)

If you decide you would like to use datetime, you should get the same results by altering your sort function to return a datetime object using datetime.strptime:

from datetime import datetime

def date_sort(filename):
    abc, hms, ymd = filename.split('_')
    return datetime.strptime(hms + ymd, ''%H%M%S%y%m%d')
user_129751400

You can use the datetime module to convert the date from strings to datetime object and sort on it

import datetime as DT

directory = os.listdir("logs")

sorted_log = sorted(directory, key=lambda x: DT.datetime.strptime(x, "abc_%H%M%S_%Y%m%d"))

sorted_log is a list of your filenames sorted according to date and time. The last two are the two most recent.

user_104260370

You can use the str.join() method and int():

import os

a, b = sorted(os.listdir("logs"), key=lambda x: int(''.join(x.split('_')[1:])))[:-2]

print(a)
print(b)

Explanation:

  1. sorted(os.listdir("logs") returns the list of files in the logs folder sorted alphabetically. That is not what we want.
  2. Insert a custom key, using the keyword argument, key. Instead of defining a whole other function, you can use the lambda method:
sorted(os.listdir("logs"), key=lambda x: x)
  1. Use x.split('_') in the lambda function to get a list of all the strings separated by a '_', and the slice [1:] to omit the base:
sorted(os.listdir("logs"), key=lambda x: x.split('_')[1:])
  1. Use the list.join() method to merge the two string in x.split('_')[1:] into a single string, and use int() to convert the resulting string into an integer:
sorted(os.listdir("logs"), key=lambda x: int(''.join(x.split('_')[1:])))
  1. Finally, use the slice [:-2] to get the two most recent files, and unpack the array returned by the slice into the two variables a and b:
a, b = sorted(os.listdir("logs"), key=lambda x: int(''.join(x.split('_')[1:])))[:-2]

Another way is to use the datetime.strptime() and datetime.strftime() methods:

import os
from datetime import datetime

base = "abc"
a, b = sorted(datetime.strptime(file, f"{base}_%H%M%S_%Y%m%d") for file in os.listdir("logs"))[:-2]

print(a.strftime(f"{base}_%H%M%S_%Y%m%d"))
print(b.strftime(f"{base}_%H%M%S_%Y%m%d"))
user_135524700
Related