Can't create S3 batch job through boto3 - gets request invalid

Viewed 2067

I am trying to create an S3 Batch (not AWS Batch, this is S3 Batch operation) job via boto3 using S3Control, but I get an "invalid request" response. I tried it through AWS S3 batch operation through the console which worked but now I am trying to do it through boto3 to create the batch job. Attached is the code and error message.

import json
import boto3
import time
import datetime
import os
s3ControlClient = boto3.client('s3control')
def lambda_handler(event, context):
    accountId = boto3.client('sts').get_caller_identity().get('Account')
    print(accountId)
    response = s3ControlClient.create_job(
        AccountId = accountId,
        ConfirmationRequired = False,
        Operation = {
            'S3PutObjectRetention': {
                'BypassGovernanceRetention': False,
                'Retention': {
                    'RetainUntilDate'   : datetime.datetime(2050, 06, 03),
                    'Mode'              : 'GOVERNANCE'
                }
            }
        },
        Report={
            'Bucket'        : 'arn:aws:s3:::test-s3-inventory',
            'Format'        : 'Report_CSV_20180820',
            'Enabled'       : True,
            #'Prefix'        : time.strftime('%Y%m%d'),
            'Prefix'        : 'report',
            'ReportScope'   : 'AllTasks'
        },
        Manifest={
            'Spec': {
                'Format': 'Report_CSV_20180820',
                'Fields': [
                    'Bucket', 'Key', 'VersionId', 'TaskStatus', 'ErrorCode', 'HTTPStatusCode', 'ResultMessage'
                ]
            },
            'Location': {
                'ObjectArn'         : 'https://test-s3-inventory.s3.amazonaws.com/job-34455-4eb5-829d-7eedrrr8564/manifest.json',
                'ETag'              : 'f4a7c0618aaed7777a5be40c266abe1f'
              }
        },
        Description = time.strftime('%Y-%m-%d')+' - Apply Retention and Legal Hold',
        Priority    = 10,
        RoleArn     = 'arn:aws:iam::277696433388194:role/s3BatchRole',
        Tags        = [
            {'Key': LOB', 'Value': 'Test'},
            {'Key': 'project',       'Value': ‘test project’}
        ]
    )

Error:
    Response:
    {
      "errorMessage": "An error occurred (InvalidRequest) when calling the CreateJob operation: Request invalid",
      "errorType": "ClientError",
      "stackTrace": [
        "  File \"/var/task/lambda_function.py\", line 13, in lambda_handler\n    response = s3ControlClient.create_job(\n",
        "  File \"/opt/python/botocore/client.py\", line 316, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n",
        "  File \"/opt/python/botocore/client.py\", line 635, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n"
      ]
    }
1 Answers

I think you have two small errors in your code:

  1. As @Limsanity82 noted, your object ARN should be in the following format:
    arn:aws:s3:::test-s3-inventory/job-34455-4eb5-829d-7eedrrr8564/manifest.json
    
  2. In your manifest spec, the formatting value (Report_CSV_20180820) is wrong; instead you want:
    S3BatchOperations_CSV_20180820
    
Related