Correct usage of ForEach -parallel in Powershell?

Question

I want to process a large number of URLs and grab the *.jpg file locations. The problem is that the $entry in the second foreach is not threadsafe. The script is firing hundreds of errors because the $entry is getting overwritten over and over. When I move the inner foreach outside of the ForEach-Object, then its working fine but very slowly. How can I process the split output properly within my ForEach-Object without getting these errors?

$array just contains a huge amount of URLs
$clean_img_array is the output array of the operation
$tmpArray is the reference to $clean_img_array in order to use it within a parallel ForEach

Errors:

InvalidOperation:
Line |
  14 |                  [void]$tmpArray:clean_img_array.Add($entry);
     |                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | You cannot call a method on a null-valued expression.

Snippet:

   $clean_img_array = [System.Collections.ArrayList]@();

   $array | ForEach-Object -Parallel {
        
        $web = Invoke-RestMethod $_;
        
        $i=1;
        foreach($entry in $web.Split("`"")){
            echo $entry;
            if($entry.IndexOf(".jpg") -ne -1 -And $entry.IndexOf("http") -ne -1){
                if($entry.IndexOf("?") -ne -1){
                    $tmpArray = $using:clean_img_array;
                    [void]$tmpArray.Add($entry.Substring(0, $entry.IndexOf('?')));
                }else{
                    $tmpArray = $using:Clean_img_array;
                    [void]$tmpArray:clean_img_array.Add($entry);
                }
                
            }
        }
        
    } -ThrottleLimit 20

user_6654942 · Answer

Here's a simple example. Both $a and $b are arrays. $b is the result of the parallel loop. It's like example 12 in the docs.

$a = 1..10
$b = $a | foreach-object -parallel { $_ + 1 }

$b

2
3
4
5
6
7
8
9
10
11

user_5441525 · Answer

Thanks for the support! I combined the answer from @js2012 with something of my own. The return alone did not solve the thread unsave behavior of $entry, but clearly untangled the workflow. But I used the inline version of .foreach with the pipe line variable $_ wich happens to be thread safe as it appears to me. Running now like a charm and also very fast for more than 2million entrys.

$array holds the URLs to be processed

$clean_img_array returns the grabbed image URLs

 $clean_img_array = $array | ForEach-Object -Parallel {

     $web = Invoke-RestMethod $_;
     $web.Split("`"").foreach({  
         if($_ -ne $null){
             if($_.IndexOf(".jpg") -ne -1 -And $_.IndexOf("http") -ne -1){
                 if($_.IndexOf("?") -ne -1){
                     return $_.Substring(0, $entry.IndexOf('?'));
                 }else{
                     return $_;
                 }           
             }
         }
     });

 } -ThrottleLimit 25

Correct usage of ForEach -parallel in Powershell?

2 Answers