Error
We have a PostgreSQL database that Azure hosts. We have two environments (staging/production) that recently encountered an error where most requests would time out. This issue is solved by restarting the back-end process or the Azure database.
This is critical for our application since the system is live and has real users.
The error does not occur consistently and seems to happen 2-3 times a week overnight.
Unfortunately, we have not succeeded in reproducing this error in local environments.
Cause
We are confident that it has to be related to our ORM framework, TypeORM. TypeORM fails silently; when a request is made to our back-end process, it receives the request, but as soon as TypeORM has to query the database, it hangs until server timeout and the query Promise seemingly never resolves. The Azure database is entirely healthy, while TypeORM cannot communicate with it, and there are no signs of any unexpected behaviour by the Azure Database.
TypeORM config
{
type: 'postgres',
host: process.env.PRODUCTION_DB_HOST,
port: +process.env.PRODUCTION_DB_PORT,
database: process.env.PRODUCTION_DB_DATABASE,
username: process.env.PRODUCTION_DB_USER,
password: process.env.PRODUCTION_DB_PASSWORD,
ssl: true,
extra: {
ssl: {
rejectUnauthorized: false,
},
},
autoLoadEntities: true,
entities: ['dist/**/*.entity{.ts,.js}'],
schema: 'public',
logging: false,
synchronize: false,
migrations: ['src/migrations/db/*{.ts}'],
cli: {
migrationsDir: 'src/migrations/db',
},
}
Version
The version we use is "@nestjs/typeorm": "8.0.2" which is a wrapper module and uses https://registry.npmjs.org/typeorm/-/typeorm-0.2.40.tgz
Solution
We want to make TypeORM fail when it cannot resolve a connection/query so we can perform error handling. Ideally, however, we want to resolve the root issue, which seems to be connection related.