ImportError: No module named 'psycopg2._psycopg'

I am using psycopg in an AWS Glue Job, where is harder to follow the instructions listed in the other answers.

What I did is installing psycopg2-binary into a directory and zip up the contents of that directory:

mkdir psycopg2-binary
cd psycopg2-binary
pip install psycopg2-binary -t  .
# in case using python3:
# python3 -m pip install --system psycopg2-binary -t  .
zip -r9 psycopg2.zip *

I then copied psycopg2.zip to an S3 bucket and add it as an extra Python library under "Python library path" in the Glue Spark job.

I then launched the job with the following script to verify if psycopg2 is present (the zip file will be downloaded by Glue into the directory in which the Job script is located)

from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
import sys
import os
import zipfile

## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

zip_ref = zipfile.ZipFile('./psycopg2.zip', 'r')
print os.listdir('.')
zip_ref.extractall('/tmp/packages')
zip_ref.close()
sys.path.insert(0, '/tmp/packages')

import psycopg2
print(psycopg2.__version__)

job.commit()

Download the compiled version of psycopg2 from this link https://github.com/jkehler/awslambda-psycopg2. As psycopg2 is C library for python, which need to be compiled on linux to make it work. The compile instruction also given on that link. Thanks to the https://github.com/jkehler.

This also happens to me in new Ubuntu 18.04. It is caused by missing one file _psycopg.py in the /usr/local/lib/python3.7/site-packages/psycopg2.

It is fixed by:

1. remove the old psycopg2 from your machine pip3 uninstall psycopg2.
2. download new pyscopg2 manually from the official page http://initd.org/psycopg/tarballs/PSYCOPG-2-7/psycopg2-2.7.7.tar.gz
3. tar xvf psycopg2-2.7.7.tar.gz
4. python setup.py build
5. sudo python setup.py install

I had this happen in Linux using Python 3.7. It is caused by missing one file _psycopg.cpython-37m-x86_64-linux-gnu.so in the /usr/local/lib/python3.7/site-packages/psycopg2. I downloaded _psycopg.cpython-37m-x86_64-linux-gnu.so from https://github.com/jkehler/awslambda-psycopg2/tree/master/psycopg2-3.7, and Copied this file into my anaconda lib.

I had the same error on Windows, this worked for me: pip install -U psycopg2

I had an older version installed, must have depreciated

For lambda functions on Python 3.7, I ended up using the psycopg2-binary library mentioned in these threads:

https://github.com/jkehler/awslambda-psycopg2/issues/51

Using psycopg2 with Lambda to Update Redshift (Python)

pip3 install psycopg2-binary==2.8.3

Snippet from these links:

I ended up using a different library: psycopg2-binary in my requirement.txt file and it working fine now.

solved it by using psycopg2-binary==2.8.3

I came to know that most times the WINDOWS packaging does not go fine with LAMBDA.

I faced same issue while running LAMBDA with WINDOWS installed 3rd party pscyopg2 packaging.

Solution:

step1>
I installed psycopg2 in Linux.
Copied both the directories psycopg2_binary-2.8.2.dist-info and psycopg2 from Linux to windows.

step2>
Along with source *.py, packaged with copied 3rd party dependencies psycopg2 in windows to *.zip file

step3>
Upload the file to LAMBDA - Here it goes, It runs successfully without any error.

for mac M1 users (can try for other users as well),

pip uninstall psycogp2

then

pip install psycopg2-binary

It works in my case.

Windows 10 with conda environment manager (fresh install of Django, wagtail with PostgreSQL), had the same error. Removed psycopg2

conda remove -n myenv psycopg2

it updated some packages, removed others (it also removed django, wagtail...). Then installed psycopg2 back

conda install -n myenv psycopg2

Tested it, import worked

python
>>> import psycopg2

Installed django, wagtail back. python manage.py migrate now populated PostgreSQL.

In my case, it was other site-packages that was exposed by installing pgcli, uninstalling pgcli resolved the issue for the time being.

This somehow penetrated virtualenv too.

For Mac python mycode.py migrate fixed the import issue