Django; ThreadPoolExecutor causes FATAL; sorry, too many clients

Viewed 15

I'm aware this error has already been posted but the context of my issue is different. I have tried proposed solutions without success.

I'm using django-apscheduler to schedule some management commands, one of these uses ThreadPoolExecutor to speed up an otherwise lengthy process.

Unfortunately, it does not close connections, which eventually leads to:

exception:connection to server at "localhost" (::1), port 5432 failed: FATAL: sorry, too many clients already

I'm using PostgreSQL backend, and when I run the management command directly I can see in pgAdmin that the number of database sessions reduces after execution.

However, when ran by django-apscheduler they accumulate and eventually cause FATAL: sorry, too many clients already.

I've tried calling close_old_connections() from from django.db import close_old_connections but that doesn't seem to do anything.

Can someone point me in the right direction?

This is what my management command looks like:

import concurrent
from django.core.management.base import BaseCommand

from data_model.models import DataModel

class Command(BaseCommand):
    help = "Do something interesting"

    def handle(self, *args, **kwargs):

        with concurrent.futures.ThreadPoolExecutor(10) as executor:
            list(executor.map(DataModel.objects.do_something_interesting))

I did also try adding expiration time CONN_MAX_AGE to DB config but that didn't help:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql_psycopg2",
        "NAME": env.str("POSTGRES_DB"),
        "USER": env.str("POSTGRES_USER"),
        "PASSWORD": env.str("POSTGRES_PASSWORD"),
        "HOST": env.str("POSTGRES_HOST"),
        "PORT": 5432,
        "CONN_MAX_AGE": 60
    },
}

Management command execution completed where the arrow is:

enter image description here

0 Answers
Related