Connection goes idle and disconnects Mongodb from AWS Lambda

Question

MongoDB idle problem is giving me nightmares. It fails without reason, it doesn't trigger disconnecting event or error. it just disconnects in the middle of API call.

I have tried reading blogs and old questions as well and tried it all, still the situation doesn't help at all.

This is my database connection code:

function connect () {

return mongoose.connect(process.env.MONGO_DB_URI, {
    useNewUrlParser: true,
    reconnectTries: Number.MAX_VALUE,
    reconnectInterval: 1000,
    bufferMaxEntries: 0, // and MongoDB driver buffering
    bufferCommands: false, // Disable mongoose buffering
    ha: true, // Make sure the high availability checks are on
    haInterval: 10000 // Run every 10 seconds
  })
}

if (process.env.MONGO_DB_URI) {
  connect()
}

with it I am also logging several events for the database.

db.on('connecting', () => {
  debug(`mongoose : connecting `)
})

db.on('connected', () => {
  debug(`mongoose : connected`)
})
db.on('open', () => {
  debug(`mongoose : open`)
})
db.on('disconnecting', () => {
  debug(`mongoose : disconnecting`)
})
db.on('disconnected', () => {
  debug(`mongoose : disconnected`)
})
db.on('close', () => {
  debug(`mongoose : close`)
})
db.on('reconnected', () => {
  debug(`mongoose : reconnected`)
})
db.on('error', (err) => {
  debug(`mongoose : error`, err)
})
db.on('all', () => {
  debug('mongoose: all')
})
db.on('fullsetup', () => {
  debug(`mongoose : fullsetup`)
})

Also before API call I check for the database connection from a middleware.

app.use(async (req, res, next) => {
  log.info('Ready State: ' + mongoose.connection.readyState, {
    action: 'App.ConnectionCheck'
  })
  if (mongoose.connection.readyState !== 1) {
    log.info('try to reconnect', {
      action: 'App.ConnectionCheck'
    })
    await connect()
    next()
  } else if (mongoose.connection && mongoose.connection.db) {
    await Promise.race([
      mongoose.connection.db.admin().ping(),
      new Promise((resolve, reject) => {
        setTimeout(() => {
          reject(new Error('Race condition failed'))
        }, 1000)
      })
    ]) 
    .then(async d => {
      log.info('ping successful: ', d)
      if (d.ok !== 1) {
        await connect()
      }
    })
    .catch(async e => {
      log.error('Ping Failed ', {
        error: e
      })
      await connect()
    })
    next()
  } else {
    next()
  }
})

But still sometimes a call comes in, and first query is made, but i dont get a response back and my API times out.

This is what my logs look like notice the time it takes to reconnect.

2020-08-10 16:30:55.465 (+05:00)        34c5812c-5446-4c0f-bddf-c4ca8a0117cb    INFO    ----------------- BODY END ------------
2020-08-10 16:30:55.466 (+05:00)        34c5812c-5446-4c0f-bddf-c4ca8a0117cb    INFO    Mongoose: meetings.countDocuments({ appntId: '', '$or': [ { deleted: true }, { createdAt: { '$gt': new Date("Mon, 10 Aug 2020 11:30:45 GMT") } {})
Mon, 10 Aug 2020 11:30:55 GMT timify:db-handle:index mongoose : disconnected
Mon, 10 Aug 2020 11:31:15 GMT timify:db-handle:index mongoose : connected
Mon, 10 Aug 2020 11:31:15 GMT timify:db-handle:index mongoose : reconnected
END RequestId: 34c5812c-5446-4c0f-bddf-c4ca8a0117cb
END RequestId: 34c5812c-5446-4c0f-bddf-c4ca8a0117cb
REPORT RequestId: 34c5812c-5446-4c0f-bddf-c4ca8a0117cb  Duration: 30030.13 ms   Billed Duration: 30000 ms       Memory Size: 10
Max Memory Used: 172 MB

Version:

"mongoose": "^5.4.17"
"nodejs": "12.x"
"mongodb-driver": "3.1"

I have seen another post regarding this issue, but my environment is different and that question is back from 2016.

user_6659799 · Answer

I have encountered a similar problem with running an Express app on AWS Lambda.

The error message I got was :

MongooseError: Cannot call `users.find()` before initial connection is complete if `bufferCommands = false`. Make sure you `await mongoose.connect()` if you have `bufferCommands = false`.

My current connection settings that work are

mongoose
    .connect(mongoConnectionString, {
        useNewUrlParser: true ,
        useCreateIndex: true,
        useUnifiedTopology: true,
        keepAlive : true,
        bufferMaxEntries: 0 // and MongoDB driver buffering
    }
);

With trial and error I have managed to track down the connection error to

bufferCommands: false

command and commenting it out made it work for me.

I couldn't find any reason behind this issue through the mongoose documentation about buffering options, but you can try turning buffering off globally by,

mongoose.set('bufferCommands', false);

Hope this solution helps.

EDIT : Setting bufferCommands to false globally, resulted in the same error.

Connection goes idle and disconnects Mongodb from AWS Lambda

1 Answers