I'm trying to search a directory for files by hash, then output the location,
i have 2 dirs one contains the reference set(referred to as mag its around 130GB) and one the target(referred to as "target" its around 130GB).
first, I'm creating a CSV with the hashes of every dir
$basedir = "E:\workrelated\test"
$magcsvpath = $basedir + "\mag.csv"
$targetcsvpath = $basedir + "\target.csv"
Get-FileHash -Algorithm MD5 -Path (Get-ChildItem $basedir"\mag\*.*" -Recurse -force) | export-csv $magcsvpath -Encoding UTF8 -NoTypeInformation
$FileContent = Get-Content $magcsvpath -Encoding UTF8
$FileContent[0] = $FileContent[0] -replace 'Hash', 'magHash'
$FileContent[0] = $FileContent[0] -replace 'Path', 'magPath'
$FileContent | Set-Content $magcsvpath -Encoding UTF8
Get-FileHash -Algorithm MD5 -Path (Get-ChildItem $basedir"\target\*.*" -Recurse -force) | export-csv $targetcsvpath -Encoding UTF8 -NoTypeInformation
then using this script im creating a csv file with the comperason
$basedir = "E:\workrelated\test"
$magcsvpath = $basedir + "\mag.csv"
$targetcsvpath = $basedir + "\target.csv"
$results = @()
$progress=0
$magFile = Import-csv $magcsvpath
$targetFile = Import-Csv $targetcsvpath
Write-Host "Find files in " $targetcsvpath
$targetFile | ForEach-Object -Parallel {
$filename = "empty"
$filemagname = "empty"
$filepath = "empty"
$filehash = "empty"
$findacopy = "false"
$CreationTime = "empty"
$LastWriteTime = "empty"
$LastAccessTime = "empty"
$Length = "empty"
foreach($magHash in $magFile)
{
Write-Host
Write-Host "=============================================================="
Write-Host $.Hash, "location" $.Path
Write-Host $magHash.magHash, "location" $magHash.magPath
Write-Host "=============================================================="
Write-Host
$filename = Split-Path $_.Path -leaf
$filepath = $_.Path
$filehash = $_.Hash
$findacopy = "false"
$CreationTime = Get-ItemProperty $filepath | select -expandproperty CreationTime -first 1
$LastWriteTime = Get-ItemProperty $filepath | select -expandproperty LastWriteTime -first 1
$LastAccessTime = Get-ItemProperty $filepath | select -expandproperty LastAccessTime -first 1
$Length = ((Get-Item $filepath).length/1KB)
Write-Host $magHash.magHash, "location" $magHash.magPath
if($magHash.magHash -contains $_.Hash)
{
$findacopy = "true"
Copy-Item $filepath $basedir\vault -Force
$filemagname = Split-Path $magHash.magPath -leaf
}
if($findacopy -eq "true")
{
Write-Host "its a match " $_.Hash
}
$details = @{
location_on_path = $filepath
has_a_copy = $findacopy
file_name = $filename
file_original_name = $filemagname
hash = $filehash
CreationTime = $CreationTime
LastWriteTime = $LastWriteTime
LastAccessTime = $LastAccessTime
Size_on_disk_KB = $Length
}
$results += New-Object PSObject -Property $details
}
$progress++
Write-Host $progress
Write-Host $results
}
Write-Host $progress
Write-Host $results
$results | export-csv -Path E:\workrelated\test\result.csv -Encoding UTF8 -NoTypeInformation -Append
it works just takes too much time, so I've tried using
ForEach-Object -Parallel
but nothing happens I'm clueless and by my calculations, I'm talking about 300+ hours of runtime
hope i will find my answer here thanks :)