Why am I getting pymongo.errors.AutoReconnect: connection pool paused

user_449193 0
Viewed 1824

I'm getting this error of autoreconnect, and there are around 100 connections in the logs during this call to .objects. Here is the document:

class NotificationDoc(Document):
    patient_id = StringField(max_length=32)
    type = StringField(max_length=32)
    sms_sent = BooleanField(default=False)
    email_sent = BooleanField(default=False)
    sending_time = DateTimeField(default=datetime.utcnow)

    def __unicode__(self):
        return "{}_{}_{}".format(self.patient_id, self.type, self.sending_time.strftime('%Y-%m-%d'))

and this is the query call that causes the issue:

    docs = NotificationDoc.objects(sending_time__lte=from_date)

This is the complete stacktrace. How can I understand what is going on?

pymongo.errors.AutoReconnect: connection pool paused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/app/src/reminder/configurations/app/worker.py", line 106, in schedule_recall
    schedule_recall_use_case.execute()
  File "/app/src/reminder/domain/notification/recall/use_cases/schedule_recall.py", line 24, in execute
    notifications_sent = self.notifications_provider.find_recalled_notifications_for_date(no_recalls_before_date)
  File "/app/src/reminder/data_providers/database/odm/repositories.py", line 42, in find_recalled_notifications_for_date
    if NotificationDoc.objects(sending_time__lte=from_date).count() > 0:
  File "/usr/local/lib/python3.8/site-packages/mongoengine/queryset/queryset.py", line 144, in count
    return super().count(with_limit_and_skip)
  File "/usr/local/lib/python3.8/site-packages/mongoengine/queryset/base.py", line 423, in count
    count = count_documents(
  File "/usr/local/lib/python3.8/site-packages/mongoengine/pymongo_support.py", line 38, in count_documents
    return collection.count_documents(filter=filter, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pymongo/collection.py", line 1502, in count_documents
    return self.__database.client._retryable_read(
  File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1307, in _retryable_read
    with self._secondaryok_for_server(read_pref, server, session) as (
  File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1162, in _secondaryok_for_server
    with self._get_socket(server, session) as sock_info:
  File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1099, in _get_socket
    with server.get_socket(
  File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1371, in get_socket
    sock_info = self._get_socket(all_credentials)
  File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1436, in _get_socket
    self._raise_if_not_ready(emit_event=True)
  File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1407, in _raise_if_not_ready
    _raise_connection_failure(
  File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 250, in _raise_connection_failure
    raise AutoReconnect(msg) from error
pymongo.errors.AutoReconnect: mongo:27017: connection pool paused
pymongo mongoengine python
2 Answers

turned out to be celery is leaving connections open, so here is the solution

from mongoengine import disconnect
from celery.signals import task_prerun

@task_prerun.connect
def on_task_init(*args, **kwargs):
    disconnect(alias='default')
    connect(db, host=host, port=port, maxPoolSize=400, minPoolSize=200, alias='default')
user_4491930

This could be related to the fact that PyMongo is not fork-safe. If you're using a process pool in any way, which includes server software like uWSGI and even some configurations of popular ASGI servers.

Every time you run a query in PyMongo, that MongoClient object becomes fork-unsafe. A MongoClient object that has never run any queries is fork-safe. Created Database and Collection objects will reference back to their parent MongoClient without checking for existence.

I do see you are using MongoEngine, which uses PyMongo underneath. I don't know the semantics of that particular library but I would assume they are the same.

In short: you must re-create your connections as part of your forking process.

user_980290
Related