Manually invoke lambda with s3 event in java

user_10631371 0
Viewed 810

Have a lambda which implements RequestHandler<S3Event, Void> and has handler as such Void handleRequest(S3Event event, Context context) {log.info(" Event => " + event)....}

S3Event here is of type com.amazonaws.services.lambda.runtime.events.S3Event;

Trying to manually invoke the lambda without uploading any data to s3. So using LambdaInvokerFactory to create lambda client and lambda interface as:

public interface Lambdas {

        @LambdaFunction(functionName = "Foo)
        Void bar(S3Event event);
    }

Manually created S3EventNotification.S3EventNotificationRecord ( of type com.amazonaws.services.lambda.runtime.events.S3EventNotification) with this reference. Logging locally the S3Event after creation prints as such :

{
    "Records": [
        {
            "records": [
                {
                    "awsRegion": "us-west-2",
                    "eventName": "ObjectCreated:Put",
                    "eventSource": "aws:s3",
                    "eventTime": {
                        "era": 1,
                        "dayOfYear": 71,
                        "year": 2021,
                        "dayOfMonth": 12,
                        "dayOfWeek": 5,
                        "weekyear": 2021,
                        "weekOfWeekyear": 10,
                        "monthOfYear": 3,
                        "millisOfDay": 0,
                        "yearOfCentury": 21,
                        "centuryOfEra": 20,
                        "millisOfSecond": 0,
                        "yearOfEra": 2021,
                        "secondOfMinute": 0,
                        "secondOfDay": 0,
                        "minuteOfHour": 0,
                        "minuteOfDay": 0,
                        "hourOfDay": 0,
                        "millis": 1615536000000,
                        "zone": {
                            "fixed": false,
                            "uncachedZone": {
                                "fixed": false,
                                "cachable": true,
                                "id": "America/Los_Angeles"
                            },
                            "id": "America/Los_Angeles"
                        },
                        "chronology": {
                            "zone": {
                                "fixed": false,
                                "uncachedZone": {
                                    "fixed": false,
                                    "cachable": true,
                                    "id": "America/Los_Angeles"
                                },
                                "id": "America/Los_Angeles"
                            }
                        },
                        "afterNow": false,
                        "beforeNow": true,
                        "equalNow": false
                    },
                    "eventVersion": "2.0",
                    "requestParameters": {
                        "sourceIPAddress": "127.0.0.1"
                    },
                    "responseElements": {
                        "xAmzId2": "Foo",
                        "xAmzRequestId": "Bar"
                    },
                    "s3": {
                        "configurationId": "Foo",
                        "bucket": {
                            "name": "Foo",
                            "ownerIdentity": {
                                "principalId": "arn:aws:iam::Foo"
                            },
                            "arn": "arn:aws:s3::Foo"
                        },
                        "object": {
                            "key": "FooBar.txt",
                            "size": 10,
                            "eTag": "Foo",
                            "versionId": "Foo",
                            "sequencer": "Foo",
                            "urlDecodedKey": "FooBar.txt",
                            "sizeAsLong": 10
                        },
                        "s3SchemaVersion": "1.0"
                    },
                    "userIdentity": {
                        "principalId": "arn:aws:iam::Foo"
                    }
                }
            ]
        }
    ]
}

But upon invoking bar(S3Event), it invokes the lambda but logs show it as empty:

Event => {[]}

Things I tried

  • Change the interface's input to Void bar(Map<String, Object> event) and invoked lambda as such bar(ImmutableMap.of("Records", event.getRecords()) It invokes the lambda but following err:
An error occurred during JSON parsing: java.lang.RuntimeException
java.lang.RuntimeException: An error occurred during JSON parsing
Caused by: lambdainternal.util.ReflectUtil$ReflectException: java.lang.reflect.InvocationTargetException
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
Caused by: java.lang.IllegalArgumentException: Invalid format: "{"year":2021,"dayOfYear":71,"equ..."
    at org.joda.time.format.DateTimeFormatter.parseDateTime(Unknown Source)
    at org.joda.time.DateTime.parse(Unknown Source)
    at org.joda.time.DateTime.parse(Unknown Source)
    at com.amazonaws.services.s3.event.S3EventNotification$S3EventNotificationRecord.<init>(S3EventNotification.java:386)
    at com.amazonaws.services.s3.event.S3EventNotification$S3EventNotificationRecord.<init>(S3EventNotification.java:355)
    ... 4 more

This is how records are been created :

return new S3EventNotification.S3EventNotificationRecord(
            region,
            EVENT_NAME,
            EVENT_SOURCE,
            DateTime.now().toString(),
            EVENT_VERSION,
            new S3EventNotification.RequestParametersEntity(SOURCE_IP_ADDRESS),
            new S3EventNotification.ResponseElementsEntity(
                RandomStringUtils.randomAlphanumeric(SIZE),
                RandomStringUtils.randomAlphanumeric(SIZE)
            ),
            s3Entity,
            getUserIdentityEntity(accountId)

Not sure what is going wrong here ?

aws-lambda amazon-web-services amazon-s3 java
3 Answers

Format of S3 event source to lambda is documented here

It is an object with array of Records

So, it should be below, we don't need a second layer of records array.

{
  "Records": [
    {
      "eventVersion": "2.1",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-2",
      "eventTime": "2019-09-03T19:37:27.192Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "AWS:AIDAINPONIXQXHT3IKHL2"
      },
      "requestParameters": {
        "sourceIPAddress": "205.255.255.255"
      },
      "responseElements": {
        "x-amz-request-id": "D82B88E5F771F645",
        "x-amz-id-2": "vlR7PnpV2Ce81l0PRw6jlUpck7Jo5ZsQjryTjKlc5aLWGVHPZLj5NeC6qMa0emYBDXOo6QBU0Wo="
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "828aa6fc-f7b5-4305-8584-487c791949c1",
        "bucket": {
          "name": "lambda-artifacts-deafc19498e3f2df",
          "ownerIdentity": {
            "principalId": "A3I5XTEXAMAI3E"
          },
          "arn": "arn:aws:s3:::lambda-artifacts-deafc19498e3f2df"
        },
        "object": {
          "key": "b21b84d653bb07b05b1e6b33684dc11b",
          "size": 1305107,
          "eTag": "b21b84d653bb07b05b1e6b33684dc11b",
          "sequencer": "0C0F6F405D6ED209E1"
        }
      }
    }
  ]
}
user_101868080

Couple things here,

  1. As Balu Vyamajala mentioned to match the records format.
  2. Was using 3.5 as dependency which created eventTime field as an object instead of string which threw com.amazonaws.services.lambda.invoke.LambdaFunctionException: Invalid format: "{"year":2019,"dayOfYear":246,"eq... upon invoking lambda. Using 2.8 created eventTime:2019-09-03T19:37:27.192Z which solved the issue.

the current interface.

@LambdaFunction(functionName = Foo)
        Void Bar(S3Event payload);

However, S3EventNotificationRecord constructor is deprecated. You can create json doc in such format and use S3EventNotification.parseJson to get the records.

user_106313710

When unit testing Lambda functions in Java I recommend to use the aws-lambda-java-tests module for JUnit.

Lambda has specific serialization logic which it implements which your own test code might not.

By using this testing library you will use the same serialisation logic as Lambda.

user_137559700
Related