I'm currently running multiple instances of a Compose service called app (--scale app=40: the computer has 56 CPU cores and 256 GB RAM) which embeds a Python script connecting to read/write data to a unique PostgreSQL database, known as a db service (this service is run only once, and is started prior to the app service up to the point postgres tells me that the database is ready to accept connections) from the same composition.
Sometimes (rarely, but this is still too much for me), this following psycopg2 error shows up:
psycopg2.OperationalError:
could not translate host name "db" to address:
Temporary failure in name resolution
Here's the full stack trace:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 310, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
rec = pool._do_get()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
self._dec_overflow()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
raise exception
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
return self._create_connection()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 371, in __init__
self.__connect()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 666, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
raise exception
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 241, in <lambda>
return lambda crec: creator()
File "/app/scripts/database_connectors.py", line 112, in connector
res = psycopg2.connect(**DB_PARAMETERS)
File "/usr/local/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not translate host name "db" to address: Temporary failure in name resolution
I'm wondering why and how could I properly debug this error or figure out what exact happened, because my db service is working well and the multiple app instances can properly reach it 99% of the time.
This does not pop up at the same time, so maybe it's not related, but just in case it could, this error also shows up from time to time:
psycopg2.errors.DiskFull:
could not resize shared memory segment "/PostgreSQL.3516559362" to 146703328 bytes:
No space left on device
or:
sqlalchemy.exc.OperationalError:
(psycopg2.errors.DiskFull) could not resize shared memory segment "/PostgreSQL.3516559362" to 146703328 bytes:
No space left on device
Versioning:
$ docker --version
Docker version 20.10.18, build b40c2f6
$ docker-compose --version
Docker Compose version v2.5.0
$ dc exec app bash
root@38dffec72240:/app# python3
Python 3.9.7 (default, Sep 3 2021, 02:02:37)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg2
>>> psycopg2.__version__
'2.9.3 (dt dec pq3 ext lo64)'
More:
$ df -h
Filesystem Size Used Avail Use% Mounted on
(...)
/dev/mapper/vps-vg-root 665G 355G 277G 57% /
(...)