Python 3: Find and delete folders with duplicate files; keep folder with most files

user_7192058 0
Viewed 268

I'm making a Python 3 program to find and delete folders with duplicate files. For example, in this instance:

/dir1/dir1: file1, file2, file3
/dir1/dir2: file1, file2, file3, file4, file5
/dir2/dir1: file1, file3, file4
/dir2/dir2: file1, file2, file3, file4, file5, file6

/dir1/dir1, /dir1/dir2 and /dir2/dir1 would be deleted. Only dir2/dir2 would be kept.

However, the program should be able to deal with other situations like:

/dir2/dir1: file1, file2, file3, file4, file5, file6
/dir2/dir2: file1, file2, file3, file4, file5, file6

or this:

/dir2/dir1: file2, file3, file4, file5, file6
/dir2/dir2: file1, file2, file4, file5, file6

The result of both examples should be neither folder would be deleted and the program would issue some sort of warning so the user could examine the folders manually.

Obviously, the program needs to account for the same files having different file paths (likely by using hashes).

It might be useful to know the final program will not delete files in place. Instead it will:

  1. Recursively scan folders and export file metadata to a json file
  2. Read one or more json files to find duplicate folders, then create a deletion script that can be run later

There also will be many other features such as whitelists, blacklists, prioritizing files in one json file, scanning zip files, etc. I think I have a handle on these extra features.

My main problem is I can't figure out the best way to go about the basic de-duplication code at the heart of the program. I would appreciate some suggestions. Thanks.

Edit: I'm adding more info about where I am in my code. So far it looks like this:

  1. Create JSON files with file metadata

  2. Import dictionary from JSON file:

json_file_to_dict_1: {
    file_key: { # I use a file hash for file_key
        'hash': <hash>, # A file hash: md5 or sha256
        'path': [full_path1, full_path_2] # a list b/c same file may have multiple paths
     }
}

json_file_to_dict_2: {
    file_key: {
        'hash': <hash>,
        'path': [full_path_1, full_path_2]
    }
}
  1. Use a for loop to make set of hashes from all files in each dictionary. Make other sets that just contain hashes with multiple paths (they have duplicates within own dictionary):
json_set_1_all_files = {hash1, hash2, hash3}
json_set_2_all_files = {hash1, hash2, hash3}

json_set_1_files_with_multiple_paths = {hash1, hash2}
json_set_2_files_with_multiple_paths = {hash1, hash2}
  1. Create a new set with hashes found in both sets (files found in both dictionaries).
dupe_set = json_set_1_all_files.intersection(json_set_2_all_files)
  1. Where do I go from here? This would be easy if I just wanted to delete duplicate files, but (as you can see above) that isn't exactly what I want to do.
duplicates python-3.x python
1 Answers

For my opinion, there are 2 things you need to consider:

  • 2 files with different name but the same data (compare by Hash and size)
  • 2 files with the same name but different data

In this situation, I have some ideas:

  • You should use os.walk in python to read the directory recursively.
  • You should use hashlib to compare 2 files and compare the size of 2 files too.
  • You should print out the warning and let user do what they want:
    • Replace all
    • Replace one by one
    • Keep all files with different names but same data
    • ...

I update for your new edit:

In here, you don't clear about the file_key in your json. But I suggest you that create a dict, and the key is the file_hash and value is a list of file_path that have the same file_hash.

Then you can save that hash into the json file.

What you need is:

json_file_to_dict_1= {
    "file_key": [full_path1, full_path_2] # a list b/c same file may have multiple paths
}

json_file_to_dict_2= {
    "file_key": [full_path_1, full_path_2]
}

When you want to drop all duplicate files in 2 difference tree, you need to loop all the key of the first dict, pop that key in the second dict and add 2 array. Then move all the rest of keys in the second dict to the first dict.

Finally, you just need to remove all the items in the list, except for the first one.

user_67573840
Related