Get last n folder name

adi bahadur 0
Viewed 90

I'm working on a python project that will frequently loop to detect any new folder created in a directory and insert the new folder name into database.

This is shape of the directory and the folders inside it:

C:\USERS\DESKTOP\RESULT
├───20220819_160033
├───20220819_162117
├───20220819_192122
├───20220820_080920
├───20220820_094355
├───20220902_081328
└───20220902_083015

The current algorithm:

I retrieve the previous folder count that have been saved in the database and compared it with current folder counts.

Here's the code:

import pandas as pd
import pyodbc, os

# Connect to SQL Server
conn = pyodbc.connect('Driver={SQL Server};'
                    'Server=server;'
                    'Database=folder;'
                    'Trusted_Connection=yes;')
cursor = conn.cursor()

cursor.execute("SELECT * FROM FileLog ORDER BY FileID DESC")
prevFileCount = cursor.fetchone()
fileCount = prevFileCount[3] # <--- retrieve previous filecounts

name = []
count = 0

dir_path = r'C:\USERS\DESKTOP\RESULT'
# Iterate directory
for path in os.listdir(dir_path): # Calculate current folder counts
    # check if current path is a file
    if os.path.isdir(os.path.join(dir_path, path)):
        name.append(os.path.join(dir_path, path))
        count += 1

The idea is, if current counts is more than previous counts, the system will take the total n different and using it to get the last n folder name and save it into database. I found this help but how can I get last n folder name instead of only the last.

python
1 Answers

If I understood correctly, at the core, what you're asking is: How to fetch directory names excluding up to n-th oldest. - n being the number in the DB.

While I think it might be better to utilize watchdog in that case - Storing any new directories in list and add to DB periodically.

But if situation not allows, such as accounting for new directory while script was offline - Maybe itertools.islice and pathlib.Path helps.

Following code is expecting file names are corresponding to created date as shown in your example.

import itertools
import pathlib

DIR_PARENT = pathlib.Path("parent_path")

DIR_COUNT = 5  # pretend we fetched recorded directory count

# prepare directory literator, filtering out files.
dir_iter = (path for path in DIR_PARENT.iterdir() if path.is_dir())

# drop first n items from directory literator, won't consume much mem.
new_dirs = [path_ for path_ in itertools.islice(dir_iter, DIR_COUNT, None)]

# now do whatever you want to do with list
# for i.e. printing names
for p in new_dirs:
    print(p)

# counting is just as simple
print(len(new_dirs))

Unless newly added directory list are giant, it shouldn't use much memory while iterating thru, and can operate with much ease than os.path is.

Demo

Directory:

~ $ tree
.
├── 20220819_160033
├── 20220819_162117
├── 20220819_192122
├── 20220820_080920
├── 20220820_094355
├── 20220902_081328
├── 20220902_083015
└── not_a_dir.notdir

>>> import pathlib
>>> import itertools
>>> dir_iter = (path for path in pathlib.Path("./").iterdir() if path.is_dir())
>>> new_dirs = [path_ for path_ in itertools.islice(dir_iter, 5, None)]
>>> new_dirs
[PosixPath('20220902_081328'), PosixPath('20220902_083015')]

# two newest directories fetched
jupiterbjy0
Related