How to run two Start-Job and wait for their results?

user_17460932 0
Viewed 361

I have a list of commands and I'm trying to invoke them using Start-Job each. Each command should return an array of objects. How do I wait on each command's array and combine them after all commands are done running? I am using PS 5.1

cmds = "Command-One", "Command-Two"

foreach cmd in cmds 
{
    Start-Job -ScriptBlock { cmd }
}
powershell-5.1 powershell
1 Answers

A simple example:

# Launch two jobs that each output two numbers, 
# collect all their output, and clean up once they terminate.
" 1, 2 ", " 3, 4 " | 
  ForEach-Object { Start-Job -ScriptBlock ([scriptblock]::Create($_)) } | 
    Receive-Job -Wait -AutoRemoveJob

Note how the input command strings (designed to output two numbers each) must be converted to script blocks via [scriptblock]::Create() so that they can be passed to Start-Job's -ScriptBlock parameter.

However, it is preferable and simpler to provide the commands as script block to begin with; using script-block literals ({ ... }):

{ 1, 2 }, { 3, 4 } | 
  ForEach-Object { Start-Job $_ } | 
    Receive-Job -Wait -AutoRemoveJob

If you want to separate the timing of creating the jobs from waiting for their completion / collecting their output:

$jobs = { 1, 2 }, { 3, 4 } | ForEach-Object { Start-Job $_ } 
 
# ... perform other tasks

$jobs | Receive-Job -Wait -AutoRemoveJob

All variants launch two jobs and (directly or indirectly) pipe the job objects output by Start-Job to Receive-Job, which collects their output (see bottom section for details):

  • -Wait additionally waits for both jobs to complete, thereby ensuring that all job output has been received and output

    • Without -Wait, Receive-Job collects all output produced so far, and returns right away, so that multiple calls may be necessary; optionally, you can collect output without consuming it (-Keep).

  • -AutoRemoveJob automatically cleans up the jobs afterwards, which by definition have terminated by then.

Note that there's also:

  • Wait-Job which performs waiting only, optionally for a given state (-State), and optionally with a timeout (-Timeout), necessitating a separate Receive-Job call to collect output.

  • Remove-Job which removes jobs only, optionally including their forceful termination (-Force), if they haven't completed yet.

How Receive-Job collects output:

  • Jobs never create direct output to the caller's console - all output, even to output streams other than the success stream, such as with Write-Host - are collected by Receive-Job and routed to the corresponding output streams of the caller.

  • Objects emitted by jobs to the success output stream (i.e. data output) are decorated with properties identifying the job of origin:

    • Properties containing auto-generated GUIDs identifying the job.

      • .RunspaceId
      • .PSSourceJobInstanceId (only while the job is running, seeminlgy)
      • Both change on every invocation; note that the optional, self-chosen symbolic name you may pass to Start-Job via -Name is not reflected here, so there is no obvious way to correlate output objects with the job they came from - see below.

    • Two more properties that are primarily of interest in PowerShell's remoting (whose infrastructure background jobs partially use):

      • .PSComputerName: always 'localhost' in background jobs
      • .PSShowComputerName: always $false in background jobs; a hint to the formatting system as to whether to show the value of the .PSComputerName property.

To illustrate the points above:

# This collects 42 in $dataOutput, and prints
# 'Some status message' to the display; you could suppress that with 6>$null
$dataOutput = 
  Start-Job { Write-Host 'Some status message'; 42 } |
    Receive-Job -Wait -AutoRemoveJob

# Use ConvertTo-Json to surface the additional properties the output
# object is decorated with; you'll see something like:
# { 
#   "value": 42,
#   "PSComputerName": "localhost",
#   "RunspaceId": "ba1c0710-cb55-4759-81a2-2b089edd92a7",
#   "PSShowComputerName": false,
#   "PSSourceJobInstanceId": "44422412-6bc2-4b0d-8379-ddd09b2d8e56"
# }
$dataOutput | ConvertTo-Json

If you want to collect and emit output for each job separately:

$jobs = { 1, 2 }, { 3, 4 } | ForEach-Object { Start-Job $_ } 

# Wait for all jobs to complete.
$jobs | Wait-Job

# Collect the output for each job separately.
# You'll see the following:
#   VERBOSE: Output from job {  1, 2  }
#   1
#   2
#   VERBOSE: Output from job {  3, 4  }
#   3
#   4
$jobs | ForEach-Object {
  Write-Verbose -Verbose "Output from job { $($_.Command) }"
  $_ | Receive-Job -Wait -AutoRemoveJob
}
user_453750
Related