Foreach object is more faster than foreach -parallel?

user_12899966 0
Viewed 954

I am just starting using powershell and I wonder why my parallel srcipt is more slower than my normal foreach object script?

My script for my normal foreachobject:

function Get-ADUsers {  #get all users in nested groups }

 function Get-NestedGroupUsers {
    param ( 
        [Parameter(Mandatory = $true)][String]$FileName,
        [Parameter(Mandatory = $true)][String]$searchFileURL
    )
    $storageHolder = @()
    # $storageHolder | Export-Csv -Path "C:\Users\demandx\Desktop\AD User Lists\$FileName.csv" -NoTypeInformation -Force 
    $groupList = Get-Content $searchFileURL 
    $groupList |  ForEach-Object { 
        $allusers = Get-ADUsers -GroupName $_
        $storageHolder += $allusers  
       
    }
    $storageHolder | select ParentGroup, Name, EmployeeNumber, Enabled, LastLogonDate, PasswordLastSet  |Export-Csv -Path "C:\Users\demandx\Desktop\$FileName.csv" -NoTypeInformation -Force
}

My script for foreach -parallel (I store the function inside psm1 then import here.)

Function Get-Members {
    param ( 
        [Parameter(Mandatory = $true)][String]$FileName,
        [Parameter(Mandatory = $true)][String]$searchFileURL
    )
    $groupList = Get-Content $searchFileURL 
    $storageHolder = $groupList |  ForEach-Object -Parallel {
        Import-Module -Name "C:\Users\demandx\Desktop\Get-ADUserMembers.psm1" 
        Get-ADUserMembers -GroupName $_ | Select-Object ParentGroup, Name, EmployeeNumber, Enabled, LastLogonDate, PasswordLastSet
    }   -ThrottleLimit 5
    
    $storageHolder |  Export-Csv -Path "C:\Users\demandx\Desktop\AD User Lists\$FileName.csv" -NoTypeInformation -Force

}

The script or my get-adusers (get all the users in nested groups)

 function Get-ADUsers { 
    param ( 
        [Parameter(ValuefromPipeline = $true, mandatory = $true)][String] $GroupName
    ) 
    [int]$circular = $null

    # result holder
    $resultHolder = @()
        $table = $null 
        $nestedmembers = $null 
        $adgroupname = $null     
  function Get-ADUsers { 
    param ( 
        [Parameter(ValuefromPipeline = $true, mandatory = $true)][String] $GroupName
    ) 
    [int]$circular = $null

    # result holder
    $resultHolder = @()
        $table = $null 
        $nestedmembers = $null 
        $adgroupname = $null     

        # get members of the group and member of
        $ADGroupname = get-adgroup $groupname -properties memberof, members

        # list all members as list (no headers) and save to var
        $memberof = $adgroupname | select -expand memberof 
       
        if ($adgroupname) {  
            if ($circular) { 
                $nestedMembers = Get-ADGroupMember -Identity $GroupName -recursive 
                $circular = $null 
            } 
            else { 
                $nestedMembers = Get-ADGroupMember -Identity $GroupName | sort objectclass -Descending
                # if get adgroupmember returns nothing, it uses the members for ordinary getADGroup
                if (!($nestedmembers)) {
                    $unknown = $ADGroupname | select -expand members
                    if ($unknown) {
                        $nestedmembers = @()
                        foreach ($member in $unknown) {
                            $nestedmembers += get-adobject $member
                        }
                    }
                }
            } 
            # loops through each member
            ForEach($nestedmember in $nestedmembers){ 
                # creates the properties into a custom object. 
                $Props = @{
                    Type            = $nestedmember.objectclass;
                    Name            = $nestedmember.name;
                    DisplayName     = "";
                    ParentGroup     = $ADgroupname.name;
                    Enabled         = "";
                    Nesting         = $nesting;
                    DN              = $nestedmember.distinguishedname;
                    Comment         = ""
                    EmployeeNumber  = "";
                    LastLogonDate   = "";
                    PasswordLastSet = "";
                } 
                # if member object is a user
                if ($nestedmember.objectclass -eq "user") { 
                    # saves all the properties in the table. 
                    $nestedADMember = get-aduser $nestedmember.Name -properties enabled, displayname, EmployeeNumber, LastLogonDate, PasswordLastSet
                    $table = new-object psobject -property $props 
                    $table.enabled = $nestedadmember.enabled
                    $table.name = $nestedadmember.samaccountname
                    $table.displayname = $nestedadmember.displayname
                    $table.EmployeeNumber = $nestedadmember.EmployeeNumber
                    $table.LastLogonDate = $nestedadmember.LastLogonDate
                    $table.PasswordLastSet = $nestedadmember.PasswordLastSet

                    #save all in 1 storage
                    $resultHOlder += $table | select type, name, displayname, parentgroup, nesting, enabled, dn, comment , EmployeeNumber, LastLogonDate, PasswordLastSet
                } 


                # if member object is group
                elseif ($nestedmember.objectclass -eq "group") {  
                    $table = new-object psobject -Property $props 
                    # if circular, meaning the groups member of list contains one of its members. 
                    # e.g. if group 2 is a member of group 1 and group 1 is a member of grou 2
                    if ($memberof -contains $nestedmember.distinguishedname) { 
                        $table.comment = "Circular membership" 
                        $circular = 1 
                    } 
                    # for circular output
                    #$table | select type, name, displayname, parentgroup, nesting, enabled, dn, comment 
                    #calling function itself
                    $resultHOlder += Get-ADUsers -GroupName $nestedmember.distinguishedName                             
                } 
                else { 
                    if ($nestedmember) {
                        $table = new-object psobject -property $props
                        $resultHolder += $table | select type, name, displayname, parentgroup, nesting, enabled, dn, comment, EmployeeNumber, LastLogonDate, PasswordLastSet
                    }
                } 
            } 
        } 



    return   $resultHOlder

}
function Get-NestedGroupUsers {
    param ( 
        [Parameter(Mandatory = $true)][String]$FileName,
        [Parameter(Mandatory = $true)][String]$searchFileURL
    )
    $storageHolder = @()
    # $storageHolder | Export-Csv -Path "C:\Users\demandx\Desktop\AD User Lists\$FileName.csv" -NoTypeInformation -Force 
    $groupList = Get-Content $searchFileURL #| ForEach-Object { $_ }
    $groupList |  ForEach-Object { 
        $allusers = Get-ADUsers -GroupName $_
        $storageHolder += $allusers  
       
    }
    $storageHolder | select ParentGroup, Name, EmployeeNumber, Enabled, LastLogonDate, PasswordLastSet  |Export-Csv -Path "C:\Users\demandx\Desktop\$FileName.csv" -NoTypeInformation -Force
}



        # get members of the group and member of
        $ADGroupname = get-adgroup $groupname -properties memberof, members

        # list all members as list (no headers) and save to var
        $memberof = $adgroupname | select -expand memberof 
       
        if ($adgroupname) {  
            if ($circular) { 
                $nestedMembers = Get-ADGroupMember -Identity $GroupName -recursive 
                $circular = $null 
            } 
            else { 
                $nestedMembers = Get-ADGroupMember -Identity $GroupName | sort objectclass -Descending
                # if get adgroupmember returns nothing, it uses the members for ordinary getADGroup
                if (!($nestedmembers)) {
                    $unknown = $ADGroupname | select -expand members
                    if ($unknown) {
                        $nestedmembers = @()
                        foreach ($member in $unknown) {
                            $nestedmembers += get-adobject $member
                        }
                    }
                }
            } 
            # loops through each member
            ForEach($nestedmember in $nestedmembers){ 
                # creates the properties into a custom object. 
                $Props = @{
                    Type            = $nestedmember.objectclass;
                    Name            = $nestedmember.name;
                    DisplayName     = "";
                    ParentGroup     = $ADgroupname.name;
                    Enabled         = "";
                    Nesting         = $nesting;
                    DN              = $nestedmember.distinguishedname;
                    Comment         = ""
                    EmployeeNumber  = "";
                    LastLogonDate   = "";
                    PasswordLastSet = "";
                } 
                # if member object is a user
                if ($nestedmember.objectclass -eq "user") { 
                    # saves all the properties in the table. 
                    $nestedADMember = get-aduser $nestedmember.Name -properties enabled, displayname, EmployeeNumber, LastLogonDate, PasswordLastSet
                    $table = new-object psobject -property $props 
                    $table.enabled = $nestedadmember.enabled
                    $table.name = $nestedadmember.samaccountname
                    $table.displayname = $nestedadmember.displayname
                    $table.EmployeeNumber = $nestedadmember.EmployeeNumber
                    $table.LastLogonDate = $nestedadmember.LastLogonDate
                    $table.PasswordLastSet = $nestedadmember.PasswordLastSet

                    #save all in 1 storage
                    $resultHOlder += $table | select type, name, displayname, parentgroup, nesting, enabled, dn, comment , EmployeeNumber, LastLogonDate, PasswordLastSet
                } 


                # if member object is group
                elseif ($nestedmember.objectclass -eq "group") {  
                    $table = new-object psobject -Property $props 
                    # if circular, meaning the groups member of list contains one of its members. 
                    # e.g. if group 2 is a member of group 1 and group 1 is a member of grou 2
                    if ($memberof -contains $nestedmember.distinguishedname) { 
                        $table.comment = "Circular membership" 
                        $circular = 1 
                    } 
                    # for circular output
                    #$table | select type, name, displayname, parentgroup, nesting, enabled, dn, comment 
                    #calling function itself
                    $resultHOlder += Get-ADUsers -GroupName $nestedmember.distinguishedName                             
                } 
                else { 
                    if ($nestedmember) {
                        $table = new-object psobject -property $props
                        $resultHolder += $table | select type, name, displayname, parentgroup, nesting, enabled, dn, comment, EmployeeNumber, LastLogonDate, PasswordLastSet
                    }
                } 
            } 
        } 



    return   $resultHOlder

}

Parallel result

-------------------------------------------
Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 2
Milliseconds      : 283
Ticks             : 622833415
TotalDays         : 0.000720872008101852
TotalHours        : 0.0173009281944444
TotalMinutes      : 1.03805569166667
TotalSeconds      : 62.2833415
TotalMilliseconds : 62283.3415

Non parallel Result

-------------------------------------------
Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 35
Milliseconds      : 322
Ticks             : 353221537
TotalDays         : 0.00040882122337963
TotalHours        : 0.00981170936111111
TotalMinutes      : 0.588702561666667
TotalSeconds      : 35.3221537
TotalMilliseconds : 35322.1537
parallel-processing powershell multithreading foreach
1 Answers

TLDR:

There are 3 reasons:

  1. To take full advantage of ForEach-Object -Parallel performance, the processing time of the Script Block needs to be significantly larger than the time to set up the thread and environment.
  2. The Import-Module will introduce an overhead.
  3. Both these factors individually are small, but multiply them by 1000 or a larger number, and they become big.

ForEach-Object -Parallel runs very different than a normal ForEach-Object.

First, a normal ForEach-Object runs inside your current PowerShell thread with access to all the variables, loaded memory, and pipelining. This is fine for 98% of all the jobs we run, and 1 second execution times are ok. In the 2% of times that we have a process that is super CPU intensive that maxes out a single CPU Core and runs forever, or we need to wait for responses (e.g. API requests) when other execution can take place, then -Parallel is what we need to look at.

The idea behind Parallel execution is to take advantage of your brand new AMD Ryzen™ Threadripper™ 3990X with 64 Cores/128 Threads, and split your process into separate "Jobs" that cand run across multiple CPU Cores and multiple threads at the same time. This could increase your speeds by orders of magnitude e.g. potentially 128 times faster.

To achieve this, ForEach-Object -Parallel creates a new "Job" for each script block you execute, and starts spreading the Jobs across CPU Cores for execution. This is great when you have long running CPU bound processes, but when you have very short and small Jobs you hit the crux of Parallel execution, where the setup takes more time than the actual execution. ForEach-Object -Parallel has to completely set up your environment for each "Job" you run, e.g. it has to spin up multiple new threads and multiple new PowerShell instances for every Job to run in.

To illustrate the amount of setup time needed, If we wrote "Hello World" once to the current thread it takes 1 milisecond:

PS C:\> Measure-Command { Write-Host "Hello World" }
Hello World

Seconds           : 0
Milliseconds      : 1
TotalMilliseconds : 1.9798

To run 1 single "Hello World" in Parallel takes 26 miliseconds:

PS C:\> Measure-Command { 1 | ForEach-Object -Parallel { Write-Host "Hello World" } }
Hello World

Seconds           : 0
Milliseconds      : 26
TotalMilliseconds : 26.052

That means that it spent about 25ms spinning up a new thread, and setting up the environment and 1 ms of actual work.

To write it 100 times on the currently running thread takes about 83ms:

PS C:\> Measure-Command { 1..100 | ForEach-Object { Write-Host "Hello World" } }
Hello World
...
Hello World
Hello World

Seconds           : 0
Milliseconds      : 83
TotalMilliseconds : 83.1846

Running in -Parallel with a -ThrottleLimit 5 takes 294ms:

PS C:\> Measure-Command { 1..100 | ForEach-Object -Parallel { Write-Host "Hello World" }  -ThrottleLimit 5 }
Hello World
...
Hello World
Hello World

Seconds           : 0
Milliseconds      : 294
TotalMilliseconds : 294.3205

This goes to show how running in Parallel can be bad for tiny individual operations. But on the flip side, if you have something that takes 1 second to run, you can start to see how it works better:

e.g. run 5 processes that take 1 second each. First on a single thread:

PS C:\> Measure-Command { 1..5 | ForEach-Object { Start-Sleep -Seconds 1 } }

Seconds           : 5
Milliseconds      : 46
TotalSeconds      : 5.046348
TotalMilliseconds : 5046.348

As expected, it takes just over 5 seconds. Now, in Parallel:

PS C:\> Measure-Command { 1..5 | ForEach-Object -Parallel { Start-Sleep -Seconds 1 } -ThrottleLimit 5 }

Seconds           : 1
Milliseconds      : 73
TotalSeconds      : 1.0732423
TotalMilliseconds : 1073.2423

It completes in just over a second. If the processing time takes significantly more time than the setup time, then -Parallel is useful.

Also, in your case, not only do you have extra overhead of the setup time, but loading a module (needed to set up the new environment), adds significantly more time to the ForEach-Object -Parallel version.

For example, lets import a module AzureAD inside our ForEach-Object script 5 times:

PS C:\> Measure-Command { 1..5 | ForEach-Object { Import-Module AzureAD } }

Seconds           : 0
Milliseconds      : 18
TotalSeconds      : 0.0185406
TotalMilliseconds : 18.5406

And now with ForEach-Object -Parallel:

PS C:\> Measure-Command { 1..5 | ForEach-Object -Parallel { Import-Module AzureAD } -ThrottleLimit 5 }

Seconds           : 0
Milliseconds      : 125
TotalSeconds      : 0.1256923
TotalMilliseconds : 125.6923

We can see that there is a significant difference because it has to load the module 5 times as opposed to only a single time inside the thread, then noticing that it is still loaded, and not re-loading it.

user_21500630
Related