S3PutObjectCopy Batch Operations change destination structure of output directories in .NET

Viewed 17

I'm using AWS.NET SDK and I'm trying to use S3CopyObjectOperation with CreateJob from batch operations. It works fine, I'm generating manifest on the fly with the following content:

mysourcebucket,folder/subdir/file1.txt
mysourcebucket,folder/subdir/file2.txt

Now I'm creating a job with CreateJob with the following request:

new CreateJobRequest(){
// some values ommited
Operation = new JobOperation(){
  S3PutObjectCopy = new S3CopyObjectOperation{
   StorageClass = Standard,
   TargetResource = dstBucketArn,
   TargetKeyPrefix = dstSubdir
   }
},
 Manifest = new JobManifest(){
   Spec = new JobManifestSpec(){Fields={"Bucket","Key"}, Format= JobManifestFormat.S3BatchOperations_CSV_20180820},
   Location= new JobManifestLocation(){
     ObjectArn = //manifest key,
     Etag = // manifest Etag
     }
   }
}

It is copying the files correctly under dstBucket. Output:

dstSubdir/folder/subdir/file1.txt
dstSubdir/folder/subdir/file2.txt

Is it possible to change target path somehow, so it doesn't include folder? Expected output:

dstSubdir/subdir/file1.txt
dstSubdir/subdir/file2.txt

Edit: Imageine simple scenario where I want to copy these objects and at some point copy them back into the same location. I won't be able to do that if I use TargetPrefix. In above example I'd create

targetbucket,dstSubdir/folder/subdir/file1.txt
targetbucket,dstSubdir/folder/subdir/file2.txt

And output would be in srcbucket:

srcSubdir/dstSubdir/folder/subdir/file1.txt
srcSubdir/dstSubdir/folder/subdir/file2.txt

The only solution that would work is not to use TargetPrefix at all and keep the same structure in src and dst buckets which is quite a big restriction in my case.

Ideally I'd like to pass

my-bucket,{"origKey": "object1key", "newKey": "newObject1Key"}
my-bucket,{"origKey": "object2key", "newKey": "newObject2Key"}
my-bucket,{"origKey": "object3key", "newKey": "newObject3Key"}

as presented in this example https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-invoke-lambda.html

0 Answers
Related