Powershell -parallel and thread safe usage?

Question

6 hours later and I am getting closer. Started learning Visual Studio Code vs ISE so I can easily use PowerShell 7. Learned about -Parallel, then how you can't call variables in the Script Block without using $using:. (That took forever!) Tried the -Asjob part but that doesn't seem right for this. I am basically taking a text file with 20 lines and search for matches in a 300k+ lines of a CSV.

I believe I have gotten to a part I just can't figure out. I think I need to know how to use "thread safe" objects based upon the error.

$textfile = "C:	est_delete\key_words.txt"
$loglocation = "C:	est_delete\datablob.csv"
$outputfile = "C:	est_delete	est5.csv"

New-Item -Path $outputfile

Get-Content $textfile | ForEach-Object -Parallel {
    Select-String  -Raw -Path $using:loglocation -Pattern $_ |
    Out-File -append $using:outputfile
}

Error Out-File: Line | 3 | Out-File -append $using:outputfile | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | The process cannot access the file 'C: est_delete est5.csv' because it is being used by another process.

In the ForEach-Object help, Example 11, it talks about thread safe usage and the example declares the variable = the thread safe syntax. That makes sense (kinda). Except in this case, I am wanting to eventually have a paramenter declared for $outfile which will need to be called in a tread safe manner. I am getting ahead of myself but just can't figure out how to make that thread safe call to $outfile.

Sorry for be verbose, this is my first post and just want to relay I have spent some time trying to get this. Thanks in advance!

user_712649 · Answer

Here's my preferred alternative - don't use ForEach-Object -Parallel here, it's unecessary:

Select-String -Path $loglocation -Pattern (Get-Content $textfile) | Out-File $outputfile -Append

That's it, you're done, no need to worry about concurrent write access :)

If you still insist, below are a couple of examples of how to handle this specific example.

The "thread-safe" dictionary assignment in the help file works because the data type in question (ConcurrentDictionary) is implemented with thread safety in mind.

If you want to write to a file in a thread safe manner, you'll need to maintain a lock or a mutex manually:

$fileLock = [System.Threading.ReaderWriterLockSlim]::new()

Get-Content $textfile | ForEach-Object -Parallel {
    $lock = $using:fileLock
    try{
        $lock.EnterWriteLock()
        Select-String  -Raw -Path $using:loglocation -Pattern $_ |
        Out-File -append $using:outputfile
    }
    finally{
        if($lock.IsWriteLockHeld){
            $lock.ExitWriteLock()
        }
    }
}

Alternatively, move the write operations outside the parallel loop:

Get-Content $textfile | ForEach-Object -Parallel {
    Select-String  -Raw -Path $using:loglocation -Pattern $_
} | Out-File -append $using:outputfile

Powershell -parallel and thread safe usage?

1 Answers