I'm using AWS.NET SDK and I'm trying to use S3CopyObjectOperation with CreateJob from batch operations. It works fine, I'm generating manifest on the fly with the following content:
mysourcebucket,folder/subdir/file1.txt
mysourcebucket,folder/subdir/file2.txt
Now I'm creating a job with CreateJob with the following request:
new CreateJobRequest(){
// some values ommited
Operation = new JobOperation(){
S3PutObjectCopy = new S3CopyObjectOperation{
StorageClass = Standard,
TargetResource = dstBucketArn,
TargetKeyPrefix = dstSubdir
}
},
Manifest = new JobManifest(){
Spec = new JobManifestSpec(){Fields={"Bucket","Key"}, Format= JobManifestFormat.S3BatchOperations_CSV_20180820},
Location= new JobManifestLocation(){
ObjectArn = //manifest key,
Etag = // manifest Etag
}
}
}
It is copying the files correctly under dstBucket. Output:
dstSubdir/folder/subdir/file1.txt
dstSubdir/folder/subdir/file2.txt
Is it possible to change target path somehow, so it doesn't include folder? Expected output:
dstSubdir/subdir/file1.txt
dstSubdir/subdir/file2.txt
Edit: Imageine simple scenario where I want to copy these objects and at some point copy them back into the same location. I won't be able to do that if I use TargetPrefix. In above example I'd create
targetbucket,dstSubdir/folder/subdir/file1.txt
targetbucket,dstSubdir/folder/subdir/file2.txt
And output would be in srcbucket:
srcSubdir/dstSubdir/folder/subdir/file1.txt
srcSubdir/dstSubdir/folder/subdir/file2.txt
The only solution that would work is not to use TargetPrefix at all and keep the same structure in src and dst buckets which is quite a big restriction in my case.
Ideally I'd like to pass
my-bucket,{"origKey": "object1key", "newKey": "newObject1Key"}
my-bucket,{"origKey": "object2key", "newKey": "newObject2Key"}
my-bucket,{"origKey": "object3key", "newKey": "newObject3Key"}
as presented in this example https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-invoke-lambda.html