How to iterate over folder, but only retrieve newest versions of files?

user_4822231 0
Viewed 132

I have a folder that is updated daily, with a new version of each file, following this naming scheme ['AA_06182020', 'AA_06202020', 'BTT_06182020', 'BTT_06202020', 'DC_06182020', 'DC_06202020', 'HOO_06182020', 'HOO_06202020']. The 06182020 in the file name is the date the of the file (mm/dd/yyyy), the more recent dates, obviously being the newer versions of the file. Right now I have a script (that runs daily) which iterates over every file in the folder, but I wish to get it so that only the newest version of each file is used. So far I've been able to retrieve a list of all the files, then parse the date portion of the name into a date time object and append that too a new list. I'm unsure of how to proceed from here, to make it so the list is sorted by date and only the newest versions of each file are selected for further processing?

from pathlib import Path
import pandas as pd 
import re
from datetime import datetime

me_data = (r"Path To Folder")
pathlist = Path(me_data).glob('**/*.xlsx')
fyl = []
new_fyls = []
for path in pathlist:
    # because path is object not string
    path_in_str = str(path)
    fyl.append(path.stem)
for entry in fyl:
    typ, date1 = entry.split('_')
    dt = datetime.strptime(date1,'%m%d%Y')
    new_fyls.append((entry, dt))
python-3.x list python
2 Answers

I suggest you modify your 2nd loop a bit with a dictionary. You can use the filename typ so only one date is kept (plus the filename for convinience). When you encounter a new date in the loop you compare with the previous for that file and store the recent one.

files = {}  # the dictionary
for entry in fyl:
    typ, date1 = entry.split('_')
    dt = datetime.strptime(date1, '%m%d%Y')
    if typ not in files or files[typ][0] < dt:  # datetime supports comparison 
        files[typ] = (dt, entry)

in the if statement the typ not in files checks for the first time you encounter a new file in the loop. while the other condition if it needs updating.

Lastly getting the most recent file names you need to get all the values stored and keep the second attribute each time.

 new_fyls = [row[1] for row in files.values()]

produces ['AA_06202020', 'BTT_06202020', 'DC_06202020', 'HOO_06202020'] with your example

user_80424000

You could try sorting using a lambda function, like this:

from datetime import datetime

files = ['AA_06182020', 'AA_06202020', 'BTT_06182020', 'BTT_06202020', 'DC_06182020', 'DC_06202020', 'HOO_06182020', 'HOO_06202020']
sorted_files = sorted(files, key=lambda x: datetime.strptime(x.split('_')[1], '%m%d%Y'), reverse=True)

This will produce a sorted files list with the newest files first (according to your naming convention).

user_105004240
Related