Deploy Pytidylib module in aws lambda by using Lamda Layers

user_7391182 0
Viewed 325

I am trying to deploy pytidylib python module into AWS lambda function by using layers . I have created the path as described in aws docs and created new layer. Now the code of pytidylib needs some libraries from /usr/lib but i have installed libraries in /python/lib/python3.7/site-packages/ , so to resolve this i added the path in environ PATH of aws linux server platform , but still the issue is not resolved. Below is my code :-

def lambda_handler(event, context):
"""Read  file from s3 on trigger."""
    s3 = boto3.client("s3")
    print(sys.platform)
    ld_library_path = os.environ["LD_LIBRARY_PATH"]
    print("old ld_library_path is ",ld_library_path)
    ld_library_path = ld_library_path + 
   ":/opt/python/lib/python3.7/site-packages/"
    os.environ["LD_LIBRARY_PATH"] = ld_library_path
    print("ld_library_path after set is 
    ",os.environ["LD_LIBRARY_PATH"])
  
ld_library_path after set is  /var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib:/opt/python/lib/python3.7/site-packages/

I want to understand is there any way i can make this work through some changes in code and make the pytidylib module run through layers . Below is the error:-

[ERROR] OSError: Could not load libtidy using any of these names: 
libtidy,libtidy.so,libtidy-0.99.so.0,cygtidy-0-99-0,tidylib,libtidy.dylib,tidy
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 68, in lambda_handler
document, errors = tidy_document(doc)
File "/opt/python/lib/python3.7/site-packages/tidylib/tidy.py", line 222, in tidy_document
return get_module_tidy().tidy_document(text, options)
File "/opt/python/lib/python3.7/site-packages/tidylib/tidy.py", line 234, in get_module_tidy
_tidy = Tidy()
File "/opt/python/lib/python3.7/site-packages/tidylib/tidy.py", line 99, in __init__
+ ",".join(lib_names))
aws-lambda-layers aws-lambda amazon-web-services
2 Answers

I tried to replicate your issue and for me Pytidylib layer works as expected.

This is the way I used to construct the layer, in case you want to give it a try. It involves a docker tool described in the recent AWS blog:

I created Pytidylib layer as follows:

  1. Create empty folder, e.g. mylayer.

  2. Go to the folder and create requirements.txt file with the content of

Pytidylib
  1. Run the following docker command (may adjust python version to your needs):
docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"
  1. Create layer as zip:
zip -r pytidylayer.zip python > /dev/null

  1. Create lambda layer based on pytidylayer.zip in the AWS Console. Don't forget to specify Compatible runtimes to python3.8.

  2. Add the layer to the lambda and test using the following lambda function:

import json
import  tidylib 

def lambda_handler(event, context):
    
    print(dir(tidylib))
    
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

The function executed correctly:

['PersistentTidy', 'Tidy', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'release_tidy_doc', 'sink', 'tidy', 'tidy_document', 'tidy_fragment']
user_2488230

I solved this by adding path of the tidy library (libtidy.so.5.2.0) into the environment variable of LD_LIBRARY_PATH of linux server

For me library was pre installed in ubuntu 18.04 server in /usr/lib .Copy the library from this path ,place it inside the tidylib folder create a zip, and follow the steps of lambda-layers creation .

user_73911820
Related