sqlite3 error on AWS lambda with Python 3

user_2514753 0
Viewed 8100

I am building a python 3.6 AWS Lambda deploy package and was facing an issue with SQLite.

In my code I am using nltk which has a import sqlite3 in one of the files.

Steps taken till now:

  1. Deployment package has only python modules that I am using in the root. I get the error: Unable to import module 'my_program': No module named '_sqlite3'

  2. Added the _sqlite3.so from /home/my_username/anaconda2/envs/py3k/lib/python3.6/lib-dynload/_sqlite3.so into package root. Then my error changed to:

    Unable to import module 'my_program': dynamic module does not define module export function (PyInit__sqlite3)

  3. Added the SQLite precompiled binaries from sqlite.org to the root of my package but I still get the error as point #2.

My setup: Ubuntu 16.04, python3 virtual env

AWS lambda env: python3

How can I fix this problem?

aws-lambda amazon-web-services python-3.x sqlite
8 Answers

This is a bit of a hack, but I've gotten this working by dropping the _sqlite3.so file from Python 3.6 on CentOS 7 directly into the root of the project being deployed with Zappa to AWS. This should mean that if you can include _sqlite3.so directly into the root of your ZIP, it should work, so it can be imported by this line in cpython:

https://github.com/python/cpython/blob/3.6/Lib/sqlite3/dbapi2.py#L27

Not pretty, but it works. You can find a copy of _sqlite.so here:

https://github.com/Miserlou/lambda-packages/files/1425358/_sqlite3.so.zip

Good luck!

user_13946970

TL;DR

Although not a very good approach but it works just fine. Use pickle module to dump whatever functionality you want in a separate file using you Local System. Export that file to AWS Project Directory, load functionality from file and use it.

In Long

So, here's what I tried, I had this same problem when I was trying to import stopwords from nltk.corpus but AWS doesn't let me import it, even installing it wasn't seems to be possible for me on Amazon AMI because of the same error _sqlite3 module not found. So to do that, what I tried was, using my local system I generated a file and dump stopwords into it. Here's the code

# Only for Local System

from nltk.corpus import stopwords
import pickle

stop = list(stopwords.words('English'))
with open('stopwords.pkl', 'wb') as f:
    pickle.dump(stop, f)

Now, I exported this file to AWS Directory using winscp and then used that functionality by loading it from the file on my project.

import pickle

# loading trained model
with open('stopwords.pkl', 'rb') as f:
    stop = pickle.load(f)

and it works just fine

user_134545350

From AusIV's answer, This version works for me in AWS Lambda and NLTK, I created a dummysqllite file to mock the required references. 

spec = importlib.util.spec_from_file_location("_sqlite3","/dummysqllite.py")
sys.modules["_sqlite3"] = importlib.util.module_from_spec(spec)
sys.modules["sqlite3"] = importlib.util.module_from_spec(spec)
sys.modules["sqlite3.dbapi2"] = importlib.util.module_from_spec(spec)
user_15600950

You need the sqlite3.so file (as others have pointed out), but the most robust way to get it is to pull from the (semi-official?) AWS Lambda docker images available in lambci/lambda. For example, for Python 3.7, here's an easy way to do this:

First, let's grab the sqlite3.so (library file) from the docker image:

mkdir lib
docker run -v $PWD:$PWD lambci/lambda:build-python3.7 bash -c "cp sqlite3.cpython*.so $PWD/lib/"

Next, we'll make a zipped executable with our requirements and code:

pip install -t output requirements.txt
pip install . -t output
zip -r output.zip output

Finally, we add the library file to our image:

cd lib && zip -r ../output.zip sqlite3.cpython*.so

If you want to use AWS SAM build/packaging, instead copy it into the top-level of the lambda environment package (i.e., next to your other python files).

user_13167860
Related