aws lambda - failed to find libmagic

user_4340793 0
Viewed 1950

I'm using in my lambda function the magic library to determine the file`s type. I first deployed it to a local container to check that everything works.

My DockerFile : 

FROM lambci/lambda:build-python3.8
WORKDIR /app
RUN mkdir -p .aws
COPY requirements.txt ./
COPY credentials /app/.aws/
RUN mv /app/.aws/ ~/.aws/
RUN pip install --no-cache-dir  -r requirements.txt
RUN pip install --no-cache-dir  -r requirements.txt -t "/app/dependencies/"
WORKDIR /app/dependencies
RUN zip -r  lambda.zip *

requirements.txt : 

python-magic
libmagic

In my local container when I run tests on the lambda logic everything went ok and passed (including the part that uses the magic code..).

I created a zip that contains the lambda.py code and with the python dependencies (last 3 lines in the docker file). When I upload the zip to aws and test the lambda I'm getting the following error : 

{
  "errorMessage": "Unable to import module 'lambda': failed to find libmagic.  Check your installation",
  "errorType": "Runtime.ImportModuleError"
}

As you can see, on my local container I'm using baseline image lambci/lambda:build-python3.8 that should be the same aws uses when the lambda is launching.

I tried also to add python-magic-bin==0.4.14 to the requirements.txt instead of the magic and libmagic but it didnt help either because it seems that this module is for windows.

Into the lambda.zip I put also the lambda.py which is the file that includes my lambda function : 

import boto3
import urllib.parse
from io import BytesIO
import magic

def lambda_handler(event, context):
    s3 = boto3.client("s3")
    if event:
        print("Event : ", event)
        event_data = event["Records"][0]
        file_name = urllib.parse.unquote_plus(event_data['s3']['object']['key'])
        print("getting file: {}".format(file_name))
        bucket_name = event_data['s3']['bucket']['name']
        file_from_s3 = s3.get_object(Bucket=bucket_name, Key=file_name)
        file_obj = BytesIO(file_from_s3['Body'].read())
        print(magic.from_buffer(file_obj.read(2048)))

What am I doing wrong ?

aws-lambda amazon-web-services python
3 Answers

While using filetype as suggested by other answers is much simpler, that library does not detect as many file types as magic does.

You can make python-magic work on aws lambda with python3.8 by doing the following:

  • Add libmagic.so.1 to a lib folder at the root of the lambda package. This lib folder will be automatically added to LD_LIBRARY_PATH on aws lambda. This library can be found in /usr/lib64/libmagic.so.1 on an amazon linux ec2 instance for example.
  • Create a magic file or take the one available on an amazon linux ec2 instance in /usr/share/misc/magic and add it to your lambda package.
  • The Magic constructor from python-magic takes a magic_file argument. Make this point to your magic file. You can then create the magic object with magic.Magic(magic_file='path_to_your_magic_file') and then call any function from python-magic you like on that object.

These steps are not necessary on the python3.7 runtime as those libraries are already present in aws lambda.

user_89122680

I didn't find a way to solve this issue, therefore, I decided to use a different library called filetype .

Example how to use it :

   file_type = filetype.guess(file_object)
    if file_type is not None:
        file_type = file_type.MIME

    file_object.seek(0)
    print("File type : {}".format(file_type))
    if file_type == "application/gzip":
        file_binary_content = gzip.GzipFile(filename=None, mode='rb', fileobj=file_object).read()

    elif file_type == "application/zip":
        zipfile = ZipFile(file_object)
        file_binary_content = zipfile.read(zipfile.namelist()[0])
user_43407930

Maybe someone finds it helpful. This is how I make my lambda become friends with python-magic. I used a local Ubuntu machine to create a zip-package and Python 3.7.

First, create a directory. The name doesn't matter.

mkdir lambda-package
cd lambda-package

Then, install locally python-magic. I used pip3 for it.

pip3 install python-magic -t .

Now, in your lambda-package directory create or copy (if you already have some code in your lambda) a lambda_function.py file. Be aware, Lambda expects such name. After this step you should have the following directory structure inside the lambda-package:

lambda-package
  │ lambda_function.py
  │ magic/
  │ python_magic-0.4.24.dist-info/

Now, zip the contents of this directory. Remember to zip the contents only, not the folder itself. So, inside the lambda-package run:

zip -r lambda-package.zip .

Final step. Create a new Lambda function from scratch. Make sure to choose a proper runtime. When your function is created, click Upload from -> choose Zip file and upload lambda-package.zip file.

Now you will be able to import python-magic normally, like this:

import magic

That's it!

p.s. The error failed to find libmagic appears under Python 3.8 runtime. Python 3.7 works fine.

user_84671020
Related