Why mongod.service is killed by my system?

user_8449770 0
Viewed 3661

I am using MongoDB 4.4 version on ubuntu 20.4, using 2 mongo instances on the same server. I encountered this problem:

Sep 03 10:45:25 mlnode systemd[1]: Stopping MongoDB Database Server...
Sep 03 10:45:25 mlnode systemd[1]: mongod.service: Succeeded.
Sep 03 10:45:25 mlnode systemd[1]: Stopped MongoDB Database Server.
Sep 03 10:45:25 mlnode systemd[1]: Started MongoDB Database Server.
Sep 03 10:55:46 mlnode systemd[1]: Stopping MongoDB Database Server...
Sep 03 10:55:46 mlnode systemd[1]: mongod.service: Succeeded.
Sep 03 10:55:46 mlnode systemd[1]: Stopped MongoDB Database Server.
Sep 03 10:55:46 mlnode systemd[1]: Started MongoDB Database Server.
Sep 03 11:05:05 mlnode systemd[1]: Stopping MongoDB Database Server...
Sep 03 11:05:05 mlnode systemd[1]: mongod.service: Succeeded.
Sep 03 11:05:05 mlnode systemd[1]: Stopped MongoDB Database Server.
Sep 03 11:05:05 mlnode systemd[1]: Started MongoDB Database Server.
Sep 03 11:13:20 mlnode systemd[1]: /lib/systemd/system/mongod.service:11: PIDFile= references a path below legacy directory /var/run/, updating /var/run/mongodb/mongod.pid → /run/mongodb/mongod.pid; please update the unit file accordingly.
Sep 03 11:25:47 mlnode systemd[1]: /lib/systemd/system/mongod.service:11: PIDFile= references a path below legacy directory /var/run/, updating /var/run/mongodb/mongod.pid → /run/mongodb/mongod.pid; please update the unit file accordingly.
Sep 03 11:56:47 mlnode systemd[1]: mongod.service: Main process exited, code=killed, status=6/ABRT
Sep 03 11:56:47 mlnode systemd[1]: mongod.service: Failed with result 'signal'

Wondering why the system is killing mongod.service, I don't have any clue. Then I checked my mongo logs:

{"t":{"$date":"2020-09-03T11:56:46.379+05:30"},"s":"F",  "c":"-",        "id":23083,   "ctx":"conn10155","msg":"Invariant failure","attr":{"expr":"ret","error":"UnknownError: -31803: WT_NOTFOUND: item not found","file":"src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp","line":1598}}
{"t":{"$date":"2020-09-03T11:56:46.379+05:30"},"s":"F",  "c":"-",        "id":23084,   "ctx":"conn10155","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2020-09-03T11:56:46.379+05:30"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"conn10155","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
mongodb
3 Answers

I figured out that my Linux ulimits were not set to the recommended values of mongod instance. One of the limits ( locked-in-memory ) was 65536 and in /lib/systemd/system/mongod.service LimitMEMLOCK was infinity. So I've changed that to 65536, restarted the service and it was working again.

This worked for me!

user_84497700

First, stop mongodb if you restarted:

sudo systemctl stop mongod

Next, I detected the database corruption:

sudo mongod --repair --dbpath /var/lib/mongodb

Next, run commands:

chown -R mongodb:mongodb /var/lib/mongodb
chown mongodb:mongodb /tmp/mongodb-27017.sock

Finally

sudo systemctl start mongod

From: https://sondnpt00343.medium.com/how-to-fix-mongod-service-32dbbe51a4ee

user_106436170

First, I detected the database corruption.

 sudo mongod --repair --dbpath /var/lib/mongodb

Then, I restarted the service but I faced another problem (exit-code).

Next, I ran following commands on the terminal :

chown -R mongodb:mongodb /var/lib/mongodb

chown mongodb:mongodb /tmp/mongodb-27017.sock

Finally, I restarted the mongodb service.

sudo systemctl restart mongod

And then this worked for me very well.

user_140138260
Related