How to skip unhashable (corrupt) files while md5 fingerprinting?

user_8883268 0
Viewed 173

The code below makes an md5/metadata fingerprint, but crashes on files with unknown corruption (e.g., files, that can be copied, mostly even opened, but that can not be hashed or zipped up [to disguise their corruption]).

Question: How one makes this code to skip or ignore any and all problem files and just do the rest? Imagine 1 million files on 8 TB. Otherwise I leave it running and having no real-time monitoring of progress, 2 days later I find out that nothing got hashed because a couple problem files made the code hung.

Part of the code (see full code below):

def createBasicInfoListFromDisk():

    global diskCompareListDetails, onlyFileNameOnDisk, driveLetter,walk_dir

    walk_dir = os.path.abspath(walk_dir)
    for root, subdirs, files in os.walk(walk_dir, topdown=True, onerror=None, followlinks=True ):
        for filename in files:
            file_path = os.path.join(root, filename)
            temp = file_path.split(":")
            driveLetter = temp[0]
            filePathWithoutDriveLetter = temp[1]
            fileSize = os.path.getsize(file_path)
            mod_on =  get_last_write_time(file_path)
            print('\t- file %s (full path: %s)' % (filename, file_path))
            print('FileName : {filename} is of size {size} and was modified on{mdt}'.format(filename=file_path,size=fileSize,mdt=mod_on ))

            diskCompareListDetails.append("\"" + filePathWithoutDriveLetter+"\",\""+str(fileSize) + "\",\"" + mod_on +'"')
            onlyFileNameOnDisk.append("\""+filePathWithoutDriveLetter+"\"")

    return

Error:

FileName : T:\problemtest\problemfile.doc is of size 27136 and was modified on2010-10-10 13:58:32
Traceback (most recent call last):
  File "t:\scripts\test.py", line 196, in <module>
    createBasicInfoListFromDisk()
  File "t:\scripts\test.py", line 76, in createBasicInfoListFromDisk
    mod_on =  get_last_write_time(file_path)
  File "t:\scripts\test.py", line 61, in get_last_write_time
    convert_time_to_human_readable = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(st.st_mtime))
OSError: [Errno 22] Invalid argument

Full code:

import os
import sys
import time
import datetime
import difflib
import decimal
import hashlib
from pip._vendor.distlib.compat import raw_input

csvListDetails = list()
csvCompareListDetails = list()
diskCompareListDetails = list()
onlyFileNameOnDisk = list()
addedFiles = list()
removedFiles = list()
driveLetter =""
finalFilesToChange=list()
finalFilesToDelete=list()
changedFiles=list()
csvfilewithPath="md5.csv"
import shutil
walk_dir=""

def findAndReadCSVFile(fileName):

    global csvListDetails 
    global csvCompareListDetails
    haveIgnoredLine = 0
    foundFile=0

    try :
        inputFileHandler = open(fileName,"rt",encoding='utf-8')
        update_time = get_last_write_time(fileName)
        print("\n   Found md5.csv, last updated on: %s" % update_time)
        foundFile=1

    except (OSError, IOError, FileNotFoundError):
        print("\n   md5.csv not found. Will create a new one.")
        return foundFile

    for line in inputFileHandler:
        if (haveIgnoredLine==0):
            haveIgnoredLine=1
            continue

        rowItem = line.replace("\n","").split('","')
        csvCompareListDetails.append('"' + rowItem[3]+',"'+rowItem[2]+'","' +rowItem[1]+'"')
        lineDetails = list()

        for detailNum in range (0,len(rowItem)):
            lineDetails.append('"' + (rowItem[detailNum].replace('"','')) + '"')

        csvListDetails.append(lineDetails)

    inputFileHandler.close()

    return foundFile

def get_last_write_time(filename):
    st = os.stat(filename)
    convert_time_to_human_readable = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(st.st_mtime))
    return convert_time_to_human_readable

def createBasicInfoListFromDisk():

    global diskCompareListDetails, onlyFileNameOnDisk, driveLetter,walk_dir

    walk_dir = os.path.abspath(walk_dir)
    for root, subdirs, files in os.walk(walk_dir, topdown=True, onerror=None, followlinks=True ):
        for filename in files:
            file_path = os.path.join(root, filename)
            temp = file_path.split(":")
            driveLetter = temp[0]
            filePathWithoutDriveLetter = temp[1]
            fileSize = os.path.getsize(file_path)
            mod_on =  get_last_write_time(file_path)
            print('\t- file %s (full path: %s)' % (filename, file_path))
            print('FileName : {filename} is of size {size} and was modified on{mdt}'.format(filename=file_path,size=fileSize,mdt=mod_on ))

            diskCompareListDetails.append("\"" + filePathWithoutDriveLetter+"\",\""+str(fileSize) + "\",\"" + mod_on +'"')
            onlyFileNameOnDisk.append("\""+filePathWithoutDriveLetter+"\"")

    return

def compareLogAndDiskLists():
    global addedFiles, removedFiles

    diff = difflib.unified_diff(csvCompareListDetails, diskCompareListDetails, fromfile='file1', tofile='file2', lineterm='', n=0)
    lines = list(diff)[2:]
    addedFiles = [line[1:] for line in lines if line[0] == '+']
    removedFiles = [line[1:] for line in lines if line[0] == '-']

    return

def displayInfoForUserInput():
    global finalFilesToChange, finalFilesToDelete

    changedOrNewFileCount = 0
    noLongerExistingFilesCount = 0
    totalSizeOfChange = 0

    for line in addedFiles:
        if line not in removedFiles:

            changedOrNewFileCount = changedOrNewFileCount +1

            elements =  line.replace("\n","").split('","')
            sizeOfFile= int(elements[1].replace('"',''))
            totalSizeOfChange = totalSizeOfChange + sizeOfFile
            finalFilesToChange.append(elements[0] +'"')

    for line in removedFiles:

        elements = line.split('","')
        if elements[0]+'"' not in onlyFileNameOnDisk:
            noLongerExistingFilesCount = noLongerExistingFilesCount + 1
            finalFilesToDelete.append(elements[0]+'"')

    GBModSz= decimal.Decimal(totalSizeOfChange) / decimal.Decimal('1073741824')
    print("\n   New or modified files on drive: {} (need to hash)".format(changedOrNewFileCount))
    print ("   Obsolete lines in md5.csv (files modified or not on drive): {} (lines to delete)".format(noLongerExistingFilesCount))
    print ("   {} files ({:.2f} GB) needs to be hashed.".format(changedOrNewFileCount,GBModSz))

    userInput = raw_input("\n   Proceed with hash? (Y/N, Yes/No) ")

    if (userInput.strip().upper() == "Y" or userInput.strip().upper() == "YES"):
        print("Continuing Processing...")
    else:
        print("You opted not to continue, Exiting")
        sys.exit()

    return

def processFiles(foundFile):

    if (foundFile==1):
        oldFileName = walk_dir+"/md5.csv"
        shutil.copy( oldFileName, getTargetFileName(oldFileName))

    BLOCKSIZE = 1048576*4
    global changedFiles
    for fileToHash in finalFilesToChange:
        hasher = hashlib.new('md5')
        fileToUse=driveLetter+":"+fileToHash.replace('"','')
        with open(fileToUse, 'rb') as afile:
            buf = afile.read(BLOCKSIZE)
            while len(buf) > 0:
                hasher.update(buf)
                buf = afile.read(BLOCKSIZE)

        fileDetails = list()
        fileDetails.append(hasher.hexdigest())
        fileDetails.append(get_last_write_time(fileToUse))
        fileDetails.append(os.path.getsize(fileToUse))
        fileDetails.append(fileToHash)
        changedFiles.append(fileDetails)

    return 

def getTargetFileName(oldFileName):
    targetFileName= walk_dir+"/generated_on_" + get_last_write_time(oldFileName).replace(" ","_").replace("-","").replace(":","")
    targetFileName = targetFileName + "__archived_on_" + datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
    targetFileName = targetFileName + "__md5.csv"
    return targetFileName


def writeCSVFile(fileName):
    try :
        outputFileHandler=open(fileName,"wt",encoding='utf-8')
        outputFileHandler.write("\"md5Hash\",\"LastWriteTime\",\"Length\",\"FullName\"\n")
        for details in csvListDetails:
            if details[3] in finalFilesToDelete:
                continue
            if details[3] in finalFilesToChange:
                continue
            outputFileHandler.write("{},{},{},{}\n".format(details[0],details[1],details[2],details[3]))

        for details in changedFiles:
            outputFileHandler.write("\"{}\",\"{}\",\"{}\",{}\n".format(details[0],details[1],details[2],details[3]))
        outputFileHandler.close()

    except (OSError, IOError, FileNotFoundError) as e:
        print("ERROR :")
        print("File {} is either not writable or some other error: {}".format(fileName,e))

    return

if __name__ == '__main__':

    walk_dir = raw_input("\n   Enter drive or directory to scan: ")
    csvfilewithPath=walk_dir+"/md5.csv"
    print("\n   Drive to scan: " + walk_dir)   

    foundFile = 0
    foundFile=findAndReadCSVFile(csvfilewithPath)
    createBasicInfoListFromDisk()
    compareLogAndDiskLists()
    displayInfoForUserInput()
    processFiles(foundFile)
    writeCSVFile(csvfilewithPath)

Trying this fix, no luck:

def get_last_write_time(filename):
    try:
        st = os.stat(filename)
        convert_time_to_human_readable = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(st.st_mtime))
    return convert_time_to_human_readable
    except OSError:
        pass
    return "ERROR"

def createBasicInfoListFromDisk():
python-3.x python
2 Answers

I agree with IMCoins and I'm very qurius on why except isn't catching the error.

So first thing I would do is to go to the source where the OSError is being raised and try to catch it explicity.

def get_last_write_time(filename):
   try:
      st = os.stat(filename)
      convert_time_to_human_readable = time.strftime("%Y-%m-%d %H:%M:%S",
                                                     time.localtime(st.st_mtime)
   return convert_time_to_human_readable
   except OSError:
      pass
   return "ERROR" #or whatever string you want add
user_85074190

Updated answer, for updated post.

As stated earlier, except statement with exception type specified catches everything. So, in order to do what want... I'm afraid possible answer are either :

  • To make a method that identifies corrupted files, and handles it properly.

  • Make try, except statement that encapsulate every part of your code where there could be an error.

Let me warn you about the second solution though, as sometimes, there are system errors that you do not want to avoid. I believe you should print the exception that you catch, in order to identify further problems you may encounter.

Just so you know, as you may not : your error is not in a try, except statement. Your error is in (if I copied and pasted properly in my editor) line 196, createBasicinfoListFromDisk(), then line 76, mod_on = get_last_write_time(file_path)

As you also mentioned you are using python 3.x, I suggest you are looking into the suppress function (https://docs.python.org/3/library/contextlib.html#contextlib.suppress).

I hope it helped you.

user_80037900
Related