Error when using powershell 7 foreach-object -parallel and AWS S3 module to download S3 folder content

user_18279572 0
Viewed 71

I have the following basic AWS powershell script downloading all of the content of an S3 bucket folder.

Get-S3Object -BucketName $s3BucketName -keyPrefix $keyPrefix | Read-S3Object -Folder "C:\temp\Events\"

Unfortunately, it runs too slow as the folder contains hundreds of small files and I need to build in parallel downloads.

I've tried to use foreach-object -parallel using the following script

$S3object = Get-S3Object -BucketName $s3BucketName -keyPrefix $keyPrefix
$S3object | foreach-object -parallel { Read-S3Object -Key $_ -Folder "C:\temp\Events\" }

But get the following error

[Error] ERROR: Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.Exception :Type : System.Management.Automation.ParameterBindingExceptionMessage : Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.ErrorId : AmbiguousParameterSetLine : 1Offset : 2CommandInvocation :MyCommand : Read-S3ObjectBoundParameters :Comparer : System.OrdinalIgnoreCaseComparerCount : 2Keys :Length : 3Length : 6Values :Length : 24Length : 25SyncRoot :Comparer : System.OrdinalIgnoreCaseComparerCount : 2Keys :Length : 3Length : 6Values :Length : 24Length : 25SyncRoot :Comparer : System.OrdinalIgnoreCaseComparerCount : 2Keys :Length : 3Length : 6Values :Length : 24Length : 25SyncRoot :Comparer : System.OrdinalIgnoreCaseComparerCount : 2Keys :Length : 3Length : 6Values :Length : 24Length : 25SyncRoot :Comparer : System.OrdinalIgnoreCaseComparerCount : 2Keys :Length : 3Length : 6Values :Length : 24Length : 25SyncRoot :Comparer : System.OrdinalIgnoreCaseComparerCount : 2Keys :Length : 3Length : 6Values :Length : 24Length : 25SyncRoot :Comparer : System.OrdinalIgnoreCaseComparerCount : 2Keys : …Values : …SyncRoot : …ScriptLineNumber : 1OffsetInLine : 2HistoryId : 1Line : Read-S3Object -Key $_ -Folder "C:\temp\Events"PositionMessage : At line:1 char:2+ Read-S3Object -Key $_ -Folder "C:\temp\Events"+

When I write-host the contents of $S3object I get this:

INFORMATION: Amazon.S3.Model.S3Object Amazon.S3.Model.S3Object Amazon.S3.Model.S3Object Amazon.S3.Model.S3Object

What am I doing wrong? Should I be using another method such as a job, and if so, how?

Thanks

powershell-7.0 amazon-web-services amazon-s3 powershell
1 Answers

Inside your -Parallel script block, provide $_ as input via the pipeline, analogous to what your first command does without parallelism:

$S3object | 
  ForEach-Object -Parallel { $_ | Read-S3Object -Folder "C:\temp\Events\" }

Alternatively, judging by the docs, you could have used -S3Object instead of -Key.

user_453750
Related