AWS S3Control create-job Invalid Request

Question

Good Afternoon,

I am trying to write a lambda function that will automatically trigger an S3 Batch operation so we don't have to keep going in and manually kicking them off. I'm involved with a data telemetry project that is gathering a lot of files from a source bucket and creating an inventory in another account & bucket. I have that configured with no problems.

The issue that I am running into is setting an event notification for when the manifest.checksum is delivered it needs to trigger a lambda that will then go in and create a batch job to move the files to their final resting home.

Here is my lambda

I've double checked and triple checked arn settings as well as bucket settings and they are all correct yet I continue to get 'invalid request' when testing the lambda

import json
import logging
import boto3
import os
from botocore.exceptions import ClientError

from datetime import datetime, timedelta

log = logging.getLogger(__name__)
log.setLevel(os.getenv("LOG_LEVEL", "INFO"))

for name in ['boto3', 's3control', 'botocore', 'boto']:
    logging.getLogger(name).setLevel(logging.ERROR)

clientControl = boto3.client('s3control')
    
def lambda_handler(event, context):
    log.debug('Creating job')
    
    filter_date = datetime.now() - timedelta(days=2)
    
    try:
        response = clientControl.create_job(
            AccountId = 'xxxx',
            ConfirmationRequired = False,
            Operation = {
                'S3PutObjectCopy': {
                    'TargetResource': 'arn:aws:s3:::vertexsmb-osplogs-uw2-lower-inventory',
                    'CannedAccessControlList': 'bucket-owner-full-control',
                    'MetadataDirective': 'COPY',
                    'ModifiedSinceConstraint': datetime(2021, 8, 17),
                    'RequesterPays': False,
                    'StorageClass': 'STANDARD',
                    'BucketKeyEnabled': False
                }
            },
            Report = {
                'Bucket': 'vertexsmb-osplogs-uw2-lower-inventory',
                'Format': 'Report_CSV_20180820',
                'Enabled': True,
                'ReportScope': 'FailedTasksOnly'
            },
            Manifest = {
                'Spec': {
                    'Format': 'S3BatchOperations_CSV_20180820',
                    "Fields": ['sourceBucket','destinationBucket','version','creationTimeStamp','fileFormat','fileSchema','files']
                },
                'Location': {
                    'ObjectArn': 'arn:aws:s3:::vertexsmb-osplogs-uw2-lower-inventory/vertexsmb-osplogs-uw2-lower/vertexsmb-osplogs-uw2-lower-inventory/2021-08-17T00-00Z/manifest.json',
                    'ETag': '713335a73472211e166604994d6c5db9'
                }
            },
            Priority = 10,
            RoleArn = 'arn:aws:iam::409148389496:role/osp-log-batch-user'
        )
    except ClientError as e:
        log.error(e)
        raise e

    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Any and all guidance is greatly appreciated -Bill

user_1570906 · Answer

I think you are missing out on ClientRequestToken.

import uuid

...

def lambda_handler(event, context):
    ...
    clientRequestToken = str(uuid.uuid4())
    ...
    response = clientControl.create_job(
      ...
      ClientRequestToken=clientRequestToken,
      ...
    )
    ....

I recommend defining separate objects for each of the parameters and passing them to create_job to keep the code simple and easy-to-debug. That helped me identify my mistake.

For example, something like this:

    response = client.create_job(
        AccountId=batchjob_accountid,
        ConfirmationRequired=False,
        Operation=job_operation,
        Report=report,
        ClientRequestToken=clientRequestToken,
        Manifest=manifest,
        Description='batch job triggered by lambda',
        Priority=10,
        RoleArn=batch_job_iam_role_arn,
    )

AWS S3Control create-job Invalid Request

1 Answers