AWS Lambda and S3 with R and aws.s3 package

user_3778744 0
Viewed 185

I'm trying to access an AWS S3 bucket within a lambda function with a custom docker image. I've setup a script to get a list of files with this code:

Sys.setenv("AWS_ACCESS_KEY_ID" = aws_s3_key,
           "AWS_SECRET_ACCESS_KEY" = aws_s3_secret,
           "AWS_DEFAULT_REGION" = "eu-west-2")

aws_s3_bucket_name <- "my-bucket"

f <- aws.s3::get_bucket(bucket = aws_s3_bucket_name, url_style = "path")

I've got no problem in running my script within docker (fedora locally) but, as soon as I try the same function in lambda (using the same image that I've configured in my system) I got this error:

..$ Code : chr "InvalidToken"
..$ Message : chr "The provided token is malformed or otherwise invalid." 
..$ Token-0 : chr "IQoJb3JpZ/BoNzJXuwChJ2R5p"| __truncated__ 
..$ RequestId: chr "CP2CBX9PA95G3X2X" 2021-06-15T00:24:17.848+02:00 
..$ HostId : chr "pTlkIhvhCTLPTiXwSBHL/qq7Y=" 2021-06-15T00:24:17.848+02:00 - attr(*, "headers")=List of 8 
..$ x-amz-bucket-region: chr "eu-west-2" 
..$ x-amz-request-id : chr "CP2CBX9PA95G3X2X"  
..$ x-amz-id-2 : chr "pTlkIhvhCTLPTiXwW4vMexnz/qq7Y="  
..$ content-type : chr "application/xml" 
..$ transfer-encoding : chr "chunked" 
..$ date : chr "Mon, 14 Jun 2021 22:24:17 GMT"
..$ server : chr "AmazonS3"
..$ connection : chr "close" 
... 
...

Maybe it's just my fault as I'm quite new to was-lambda but I still not get why the script works in my local docker.

I suspect that the problem is related to some encoding problem as the error I get seems to include some "\n" and the message refer to "Bad Request (HTTP 400)" :

- attr(*, "request_canonical")= chr "GET\n/xxxxxxxx-x-xxx-counter/\n\nhost:s3-eu-west-2.amazonaws.com"| __truncated__
- attr(*, "request_canonical")= chr "GET\n/xxxxxxxx-x-xxx-/\n\nhost:s3-eu-west-2.amazonaws.com"| __truncated__

- attr(*, "request_string_to_sign")= chr "AWS4-HMAC-SHA256\n" __truncated__

ERROR [2021-06-15 10:55:06] Error in parse_aws_s3_response(r, Sig, verbose = verbose): Bad Request (HTTP 400).
aws-lambda docker r amazon-s3
1 Answers

What I did was to set the aws_access_key_id & aws_secret_access_key inside the Dockerfile by using the below commands:

RUN aws configure set aws_access_key_id <<aws_access_key_id>>
RUN aws configure set aws_secret_access_key <<aws_secret_access_key>>

With this way the aws-cli credentials are being stored inside the container hence this can write to S3 and I suppose to other services as well. If you set them inside your .R script, then those are being stored inside it and the container is not aware of any AWS instance.

EDIT: A more "elegant" and secure way for the above approach would be to set arguments & envrinonment variables inside Dockerfile as per below:

ARG AWS_KEY
ARG AWS_SECRET

ENV AWS_KEY ${AWS_KEY}
ENV AWS_SECRET ${AWS_SECRET}

Then, you can use "docker build" with build arguments by specifying the keys.

docker build --build-arg AWS_KEY=xxxx --build-arg AWS_SECRET=yyyy .
user_90514280
Related