powershell ForEach-Object -Parallel on 2 csv files

Viewed 42

I'm trying to search a directory for files by hash, then output the location,

i have 2 dirs one contains the reference set(referred to as mag its around 130GB) and one the target(referred to as "target" its around 130GB).

first, I'm creating a CSV with the hashes of every dir

$basedir = "E:\workrelated\test"

$magcsvpath = $basedir + "\mag.csv"
$targetcsvpath = $basedir + "\target.csv"


Get-FileHash -Algorithm MD5 -Path (Get-ChildItem $basedir"\mag\*.*" -Recurse -force) | export-csv $magcsvpath -Encoding UTF8 -NoTypeInformation

$FileContent = Get-Content $magcsvpath -Encoding UTF8
$FileContent[0] = $FileContent[0] -replace 'Hash', 'magHash'
$FileContent[0] = $FileContent[0] -replace 'Path', 'magPath'
$FileContent | Set-Content $magcsvpath -Encoding UTF8

Get-FileHash -Algorithm MD5 -Path (Get-ChildItem $basedir"\target\*.*" -Recurse -force) | export-csv $targetcsvpath -Encoding UTF8 -NoTypeInformation

then using this script im creating a csv file with the comperason

 $basedir = "E:\workrelated\test"

$magcsvpath = $basedir + "\mag.csv"
$targetcsvpath = $basedir + "\target.csv"


$results = @()


$progress=0
$magFile = Import-csv $magcsvpath
$targetFile = Import-Csv $targetcsvpath
Write-Host "Find files in " $targetcsvpath

$targetFile | ForEach-Object -Parallel {
$filename = "empty"
$filemagname = "empty"
$filepath = "empty"
$filehash = "empty"
$findacopy = "false"
$CreationTime = "empty"
$LastWriteTime = "empty"
$LastAccessTime = "empty"
$Length = "empty"
     foreach($magHash in $magFile) 
  { 

    Write-Host
    Write-Host "=============================================================="
    Write-Host $.Hash, "location" $.Path
    Write-Host $magHash.magHash, "location" $magHash.magPath
    Write-Host "=============================================================="
    Write-Host
    $filename = Split-Path $_.Path -leaf
    $filepath = $_.Path
    $filehash = $_.Hash
    $findacopy = "false"
    $CreationTime = Get-ItemProperty $filepath | select -expandproperty CreationTime -first 1
    $LastWriteTime = Get-ItemProperty $filepath | select -expandproperty LastWriteTime -first 1
    $LastAccessTime = Get-ItemProperty $filepath | select -expandproperty LastAccessTime -first 1
    $Length = ((Get-Item $filepath).length/1KB)
    Write-Host $magHash.magHash, "location" $magHash.magPath
    if($magHash.magHash -contains $_.Hash)
    {
    $findacopy = "true"
    Copy-Item $filepath $basedir\vault -Force
    $filemagname = Split-Path $magHash.magPath -leaf
    }
   
    if($findacopy -eq "true")
    {
    Write-Host "its a match " $_.Hash
    }
    $details = @{
                location_on_path     = $filepath
                has_a_copy           = $findacopy
                file_name            = $filename
                file_original_name   = $filemagname
                hash                 = $filehash                        
                CreationTime         = $CreationTime
                LastWriteTime        = $LastWriteTime
                LastAccessTime       = $LastAccessTime
                Size_on_disk_KB      = $Length
    }
    $results += New-Object PSObject -Property $details 
   }
   $progress++
   Write-Host $progress
   Write-Host $results
}
Write-Host $progress
Write-Host $results
$results | export-csv -Path E:\workrelated\test\result.csv -Encoding UTF8 -NoTypeInformation -Append

it works just takes too much time, so I've tried using

ForEach-Object -Parallel

but nothing happens I'm clueless and by my calculations, I'm talking about 300+ hours of runtime

hope i will find my answer here thanks :)

0 Answers
Related