Python - Find duplicate files and move to another folder

user_6213331 0
Viewed 228

Working on a small script that allows the user to selected a folder to search for duplicate files, whether that be images, text etc. It should then move those duplicate files into another folder of the users choice.

This is the code i have so far:

from tkinter import Tk
from tkinter.filedialog import askdirectory

import os
import shutil

import hashlib

Tk().withdraw()

source = askdirectory(title="Select the source folder")

walker = os.walk(source)
uniqueFiles = dict()
total = 0

for folder, sub_folder, files in walker:
    for file in files:
        filepath = os.path.join(folder, file)
        filehash = hashlib.md5((open(filepath, "rb").read())).hexdigest()

        if filehash in uniqueFiles:
            print(f"{filepath} is a duplicate")
            total += 1
        else:
            uniqueFiles[filehash] = source

    print("\n# of duplicate files found: {} ".format(total))

    # destination = askdirectory(title="Select the target folder")
    # shutil.move(filepath, destination, copy_function=shutil.copytree)

It works perfectly fine for now, finding all the duplicate files in a folder/sub folders and printing them out. The part im stuck in is how to move them. the commented code at the bottom seems to work but it prompts the user for a folder for every duplicate found. I just want it to list out all the duplicates and then move them at once.

Any ideas on how i could format my code?

Thanks!

shutil filesystems python
1 Answers

So you have two options here (as described my the comments to your question):

  1. Prompt for the target directory beforehand
  2. Prompt for the target directory afterward

The first option is probably the simplest, most efficient, and requires the smallest amount of refactoring. It does however require the user to input a target directory weather or not there are any duplicate files or an error occurs when searching so might be worse from a user's perspective:

# prompt for directory beforehand
destination = askdirectory(title="Select the target folder")

for folder, sub_folder, files in walker:
    for file in files:
        filepath = os.path.join(folder, file)
        filehash = hashlib.md5(open(filepath, "rb").read()).hexdigest()

        if filehash in uniqueFiles:
            shutil.move(filepath, destination, copy_function=shutil.copytree)
        else:
            uniqueFiles[filehash] = source

The second option would allow you to perform all the necessary checks and error handling, but is more complex and requires more refactoring:

# dictionary of hashes to all files
hashes = {}

for folder, sub_folder, files in walker:
    for file in files:
        filepath = os.path.join(folder, file)
        filehash = hashlib.md5(open(filepath, "rb").read()).hexdigest()

        if filehash in hashes
            hashes[filehash].append(filepath)
        else:
            hashes[filehash] = [filepath]

# prompt for directory beforehand
destination = askdirectory(title="Select the target folder")

for duplicates in hashes.values():
    if len(duplicates) < 2:
        continue

    for duplicate in hashes:
        shutil.move(duplicate, destination, copy_function=shutil.copytree)

As a side note, I am not familiar with hashlib but I suspect that you will want to be closing the files you are hashing especially if checking a large file tree:

with open(filepath, "rb") as file:
    filehash = hashlib.md5(file.read()).hexdigest()
user_115480570
Related