Memory usage powershell 7.03 Foreach-object Parallel

user_14166612 0
Viewed 398

I am trying to do process across a reasonable data set and was trying the Foreach-Object -Parallel in powershell 7.x.

Every time i ran i was finding that i was running out of memory It seems about 1 gig per 1000 objects and i have now distilled down to the following code. (This dose nothing apart from create and then destroy 4000 runs through doing nothing)

$a = new-object object[] 4000
$a | ForEach-Object -ThrottleLimit 40 -Parallel {
  #Code to do something in here.
}

If you change the value of $a to be 1000 about 1 gig of memory will be consumed, Set $a to 2000 then 2 gig of memory etc.. The throttle limit dosnt change the amount of memory that is consumed just number of CPU threads.

This causes issues when the number of times you want to process your scriptblock gets large. In my example i need to process 20,000 computers with the code that would be in the script block. When running that code the script was consuming 16+ gig and causing paging.

Can anyone please confirm if they are seeing the same issue or if there is a known work around.

powershell-7.0 powershell
1 Answers

Thank you to Mathias R. Jessen who confirmed that this is still an issue and a suggestion. Breaking the process down to now processing only a limited number of machines to limit memory usage.


$ObjectsToProcess = (1..4102)
$Batch = 0
$BatchSize = 100
do {
    # Create an arrary with limited number of objects in it for memory management
    $ObjectsProcessing = [System.Collections.Generic.List[string]]::new()
    for ($i = $Batch; (($i -lt $Batch + $BatchSize) -and ($i -lt $ObjectsToProcess.count)); $i++) {
        $ObjectsProcessing.add($ObjectsToProcess[$i])
    }

    $ObjectsProcessing | ForEach-Object -ThrottleLimit 40 -Parallel {
        $_
        # Main script in this block now.
    }
    [gc]::Collect() # garbage collection to recover memory
    
    $Batch = $Batch + $BatchSize
} while ($Batch -lt $ObjectsToProcess.count)

This will allow processing of up to $BatchSize machines at once with the number of simultaneous machines been controlled by ThrottleLimit. Memory consumption can be controlled by $BatchSize.

Memory to now process 4102 objects now never goes above 200 Meg vs 4 Gig before.

user_141666120
Related