Flask, gunicorn, psycopg2 - OperationalError SSL error: decryption failed or bad record mac

Viewed 763

I have gunicorn running sync workers with a ThreadedConnectionPool as a singleton attribute using static methods in a class. I am running some background jobs in the master using the BackgroundScheduler of apscheduler. When a job runs on the master and a worker times out (due to a heavy database call), i start getting 500 errors on some subsequent requests on the new worker that call the DB with the error psycopg2.OperationalError: SSL error: decryption failed or bad record Mac

Here is my code.

class WriterDbConnectionPool:
    __pool = None

    @staticmethod
    def get_db_connection_pool():
        if not WriterDbConnectionPool.__pool:
            logger.info("WriterDbConnectionPool initialising connection pool")
            db_config = get_db_configs()
            WriterDbConnectionPool.__pool = ThreadedConnectionPool(1, 10,
                             database=xxx,
                             user=xxx,
                             password=xxx,
                             host=xxx,
                             port=xxx
                             )

        return WriterDbConnectionPool.__pool

    @staticmethod
    def close():
        if WriterDbConnectionPool.__pool:
            logger.info("WriterDbConnectionPool closing connection pool")
            WriterDbConnectionPool.__pool.closeall()
            WriterDbConnectionPool.__pool = None

I am running gunicorn with the following options

-k sync
--workers 3
--timeout 120
--bind 0.0.0.0:5000
--config gunicorn_config.py
--access-logfile -
--access-logformat
"%(p)s %(h)s %(l)s %(r)s %(s)s %(b)s %(f)s %(a)s %(L)s"
app:app

The code that invokes the DbConnectionPool call is below

@contextmanager
def get_db_connection():
    try:
        connection = DbConnectionPool.get_db_connection_pool().getconn()
        yield connection
    finally:
        DbConnectionPool.get_db_connection_pool().putconn(connection)

How do I solve this issue? Should i use a different worker class?

My psycopg2 version is 2.7.5, flask is 0.12.2 and gunicorn 20.0.4

1 Answers

What I did was to move the apscheduler out into a seperate python process and used the BlockingScheduler to run my background jobs.

Related