NestJS Mongoose connection dies on load testing

Question

When multiple devs use my API, multiple concurrent requests are being sent to Mongoose. When the concurrency is high, the connection just "dies" and refuses to fulfil any new request, no matter how long I wait (hours!).

I just want to state everything is working fine on regular use. Heavy use leads the connection to crash.

My MongooseModule initialization:

MongooseModule.forRoot(DatabasesService.MONGO_FULL_URL, {
  useNewUrlParser: true,
  useUnifiedTopology: true,
  useFindAndModify: false,
  autoEncryption: {
    keyVaultNamespace: DatabasesService.keyVaultNamespace,
    kmsProviders: DatabasesService.kmsProviders,
    extraOptions: {
      mongocryptdSpawnArgs: ['--pidfilepath', '/tmp/mongocryptd.pid']
    }
  } as any
})

Module that imports the feature:

@Module({
  imports: [MongooseModule.forFeature([{ name: 'modelName', schema: ModelNameSchema }])],
  providers: [ModelNameService],
  controllers: [...],
  exports: [...]
})

Service:

@Injectable()
export class ModelNameService {
  constructor(
    @InjectModel('modelName') private modelName: Model
  ) {}

  async findAll(): Promise {
    const result: IModelName[] = await this.modelName.find().exec();
    if (!result) throw new BadRequestException(`No result was found.`);
    return result;
  }
}

I've tried loadtesting using different utils, the easiest was:

ab -c 200 -n 300 -H "Authorization: Bearer $TOKEN" -m GET -b 0 https://example.com/getModelName

Any new request after the connection hangs gets stuck at ModelNameService.findAll() first line (the request to mongo).

On mongodb logs with verbosity of "-vvvvv" I can see few suspicious lines:

User Assertion: Unauthorized: command endSessions requires authentication src/mongo/db/commands.cpp

Cancelling outstanding I/O operations on connection to 127.0.0.1:33134

And I've also found that it doesn't exceed 12 open connections at the same time. It always waits to close one before opening a new one.

Other key points:

Mongoose doesn't return any value or notifies about any error. It just hangs without notifying anything.
Terminus health check able to ping the DB and returns a healthy status.
NestJS API still works - I'm able to send new requests and receive a response. Just requests that are related to the faulty connection hang.
When I inject connection and check its readyState it returns connected.
Restarting the API fixes it immediately.
MongoDB itself keeps working as normal.
Increasing Mongoose poolSize is able to handle more requests at the same time but will still crash on a larger amount of requests.

My main question here is how do I handle this case? Currently, I've added another health check to try and send a query to the problematic connection every half a minute, and k8s restarts the pod if it determines a failure. This works but it is not optimal.

NestJS Mongoose connection dies on load testing

0 Answers