psycopg2.OperationalError errors

user_3868134 0
Viewed 2300

I have created a Python flask web app and deployed it on an Azure App service using gunicorn. The web app uses flask_sqlalchemy to connect to a PostgreSQL database which is also deployed on an Azure Database for PostgreSQL server.

I would sometimes get the error below after the web app inserts a "small" query of one row with six fields:

sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). 
Original exception was: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected

[SQL: INSERT INTO <table>]
[parameters: <row's parameters>]
(Background on this error at: http://sqlalche.me/e/e3q8) (Background on this error at: http://sqlalche.me/e/7s2a)

Whilst the web app was still deployed, I restarted the database server and found the drop in connection leads to the error. If I then restart the app service, the issue gets resolved.

When I locally deploy the web app and disconnect the database I get a slightly different error:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.

[SQL: INSERT INTO <table>]
[parameters: <row's parameters>]
(Background on this error at: http://sqlalche.me/e/e3q8)

Following advice from SQLAlchemy, I changed the database connection to:

db = SQLAlchemy(engine_options={"pool_pre_ping": True})

This solved the problem for the locally deployed web app, but the same error message still appears for the Azure deployed web app.

Any advice?

pip freeze:

aniso8601==8.0.0
atomicwrites==1.3.0
attrs==19.3.0
certifi==2019.11.28
cffi==1.14.0
chardet==3.0.4
Click==7.0
colorama==0.4.3
cryptography==2.8
Flask==1.1.1
flask-restx==0.1.1
Flask-SQLAlchemy==2.4.1
idna==2.8
importlib-metadata==1.5.0
itsdangerous==1.1.0
Jinja2==2.11.1
jsonschema==3.2.0
jwt==0.6.1
MarkupSafe==1.1.1
more-itertools==8.2.0
numpy==1.18.1
packaging==20.1
pandas==1.0.1
pluggy==0.13.1
psycopg2==2.8.4
py==1.8.1
pycparser==2.19
PyJWT==1.7.1
pyparsing==2.4.6
pyrsistent==0.15.7
pytest==5.3.5
python-dateutil==2.8.1
pytz==2019.3
requests==2.22.0
scipy==1.4.1
six==1.14.0
SQLAlchemy==1.3.13
urllib3==1.25.8
wcwidth==0.1.8
Werkzeug==0.16.1
wincertstore==0.2
zipp==3.0.0
flask-sqlalchemy flask azure python
1 Answers

We fixed it by having more defensive connections to the database; putting all connections in a try/catch, and if a psycopg2 exception was raised, flushing and retrying periodically.

attempts = 0
while attempts <= 5:
    attempts = attempts + 1
    logger.info("Attempt " + str(attempts))
    try:
        db.session.add(job_status)
        db.session.commit()
        return
    except (SQLAlchemyError, OperationalError) as e:
        if attempts < 5:
            logger.warning("Attempt failed. Trying rollback")
            db.session.rollback()
            time.sleep(5)
        else:
            logger.error("Maximum number of retries reached. Raising an error")
            raise e

Answer taken from an update to the question by AmyChodorowski.

user_53209060
Related