How to use gnupg in an AWS Lambda function

user_387774 0
Viewed 4120

I have a local python code which GPG encrypts a file. I need to convert this to AWS Lambda, once a file has been added to AWS S3 which triggers this lambda.

My local code

import os
import os.path
import time
import sys
gpg = gnupg.GPG(gnupghome='/home/ec2-user/.gnupg')

path = '/home/ec2-user/2021/05/28/'
ptfile = sys.argv[1]

with open(path + ptfile, 'rb')as f:
        status = gpg.encrypt_file(f, recipients=['user@email.com'], output=path + ptfile + ".gpg")

print(status.ok)
print(status.stderr)

This works great when I execute this file as python3 encrypt.py file.csv and the result is file.csv.gpg

I'm trying to move this to AWS Lambda and invoked when a file.csv is uploaded to S3.

import json
import urllib.parse
import boto3
import gnupg
import os
import os.path
import time

s3 = boto3.client('s3')

def lambda_handler(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
    
    try:
        gpg = gnupg.GPG(gnupghome='/.gnupg')
        ind = key.rfind('/')
        ptfile = key[ind + 1:]
        with open(ptfile, 'rb')as f:
            status = gpg.encrypt_file(f, recipients=['email@company.com'], output= ptfile + ".gpg")
        print(status.ok)
        print(status.stderr)

My AWS Lambda code zip created a folder structure in AWS enter image description here

The error I see at runtime is [ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'gnupg' Traceback (most recent call last):

aws-lambda python-3.8 gnupg amazon-s3 python
5 Answers

You can create a gpg binary suitable for use by python-gnupg on AWS Lambda from the GnuPG 1.4 source. You will need

  • GCC and associated tools (sudo yum install -y gcc make glibc-static on Amazon Linux 2)
  • pip
  • zip

After downloading the GnuPG source package and verifying its signature, build the binary with

$ tar xjf gnupg-1.4.23.tar.bz2
$ cd gnupg-1.4.23
$ ./configure
$ make CFLAGS='-static'
$ cp g10/gpg /path/to/your/lambda/

You will also need the gnupg.py module from python-gnupg, which you can fetch using pip:

$ cd /path/to/your/lambda/
$ pip install -t . python-gnupg

Your Lambda’s source structure will now look something like this:

.
├── gnupg.py
├── gpg
└── lambda_function.py

Update your function to pass the location of the gpg binary to the python-gnupg constructor:

gpg = gnupg.GPG(gnupghome='/.gnupg', gpgbinary='./gpg')

Use zip to package the Lambda function:

$ chmod o+r gnupg.py lambda_function.py 
$ chmod o+rx gpg
$ zip lambda_function.zip gnupg.py gpg lambda_function.py
user_50131250

Since there are some system dependencies required to use gpg within python i.e gnupg itself, you will need to build your lambda code using the container runtime environment: https://docs.aws.amazon.com/lambda/latest/dg/lambda-images.html

Using docker will allow you to install underlying system dependencies, as well as import your keys.

Dockerfile will look something like this:

FROM public.ecr.aws/lambda/python:3.8

RUN apt-get update && apt-get install gnupg

# copy handler file
COPY app.py <path-to-keys> ./

# Add keys to gpg
RUN gpg --import <path-to-private-key>
RUN gpg --import <path-to-public-key>

# Install dependencies and open port
RUN pip3 install -r requirements.txt

CMD ["app.lambda_handler"]

app.py would be your lambda code. Feel free to copy any necessary files besides the main lambda handler.

Once the container image is built and uploaded. The lambda can now use the image (including all of its dependencies). The lambda code will run within the containerized environment which contains both gnupg and your imported keys.

Resources:

https://docs.aws.amazon.com/lambda/latest/dg/python-image.html https://docs.aws.amazon.com/lambda/latest/dg/lambda-images.html https://medium.com/@julianespinel/how-to-use-python-gnupg-to-decrypt-a-file-into-a-docker-container-8c4fb05a0593

user_107453930

gpg is now already installed in public.ecr.aws/lambda/python:3.8.

However despite that does not seem to be available from Lambda. So you still need to get the gpg executable into the Lambda environment.

I did it using a docker image.

My Dockerfile is just:

FROM public.ecr.aws/lambda/python:3.8

COPY .venv/lib/python3.8/site-packages/ ./
COPY test_gpg.py .

CMD ["test_gpg.lambda_handler"]

.venv is the directory with my python virtualenv containing the python packages I need.

user_167186660

The best way to do this is to add a lambda layer to your python lambda.

You need to make a virtual environment in which you pip install gnupg and then put all the installed python packages in a zip file, which you upload to aws as a lambda layer. This lambda layer can then be used in all lambdas where you need gnupg. To create the lamba layer you basically do:

python3.9 -m venv my_venv
 ./my_venv/bin/pip3.9 install gnupg
cp -r ./my_venv/lib/python3.9/site-packages/ python
zip -r lambda_layer.zip python

Where the python version above has to match that of the python function in your lambda.

If you don't want to use layers you can additionally do:

zip -r lambda_layer.zip ./.gnupg
zip lambda_layer.zip lambda_funtion.py

And you get a zip file that you can use as a lambda deployment package

user_8938570

The python-gnupg package requires you to have a working installation of the gpg executable, as mentioned in their official docs' Deployment Requirements; I am yet to find a way to access a gpg executable from lambda.

user_96369420
Related