psycopg2 in compose service could sometimes (<1%) not translate host name "db" to address: Temporary failure in name resolution

Viewed 40

I'm currently running multiple instances of a Compose service called app (--scale app=40: the computer has 56 CPU cores and 256 GB RAM) which embeds a Python script connecting to read/write data to a unique PostgreSQL database, known as a db service (this service is run only once, and is started prior to the app service up to the point postgres tells me that the database is ready to accept connections) from the same composition.

Sometimes (rarely, but this is still too much for me), this following psycopg2 error shows up:

psycopg2.OperationalError:
  could not translate host name "db" to address:
    Temporary failure in name resolution

Here's the full stack trace:

 Traceback (most recent call last):
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect
     return fn()
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 310, in connect
     return _ConnectionFairy._checkout(self)
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
     fairy = _ConnectionRecord.checkout(pool)
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
     rec = pool._do_get()
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
     self._dec_overflow()
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
     compat.raise_(
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
     raise exception
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
     return self._create_connection()
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection
     return _ConnectionRecord(self)
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 371, in __init__
     self.__connect()
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 666, in __connect
     pool.logger.debug("Error on connect(): %s", e)
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
     compat.raise_(
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
     raise exception
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
     self.dbapi_connection = connection = pool._invoke_creator(self)
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 241, in <lambda>
     return lambda crec: creator()
   File "/app/scripts/database_connectors.py", line 112, in connector
     res = psycopg2.connect(**DB_PARAMETERS)
   File "/usr/local/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
 psycopg2.OperationalError: could not translate host name "db" to address: Temporary failure in name resolution

I'm wondering why and how could I properly debug this error or figure out what exact happened, because my db service is working well and the multiple app instances can properly reach it 99% of the time.

This does not pop up at the same time, so maybe it's not related, but just in case it could, this error also shows up from time to time:

psycopg2.errors.DiskFull:
  could not resize shared memory segment "/PostgreSQL.3516559362" to 146703328 bytes:
    No space left on device

or:

sqlalchemy.exc.OperationalError:
  (psycopg2.errors.DiskFull) could not resize shared memory segment "/PostgreSQL.3516559362" to 146703328 bytes:
    No space left on device

Versioning:

$ docker --version
Docker version 20.10.18, build b40c2f6

$ docker-compose --version
Docker Compose version v2.5.0

$ dc exec app bash

root@38dffec72240:/app# python3

Python 3.9.7 (default, Sep  3 2021, 02:02:37) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import psycopg2
>>> psycopg2.__version__
'2.9.3 (dt dec pq3 ext lo64)'

More:

$ df -h
Filesystem                              Size  Used Avail Use% Mounted on
(...)
/dev/mapper/vps-vg-root                 665G  355G  277G  57% /
(...)
0 Answers
Related