MongoDb (k8s pod) is failing due to Operation is not permitted

Viewed 125

I'm running a mongoDB (5.0.12) instance as a kubernetes pod. Suddenly the pod is failing and I need some help to understand the logs:

{"t":{"$date":"2022-09-13T18:39:51.104+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"AuthorizationManager-1","msg":"WiredTiger error","attr":{"error":1,"message":"[1663094391:104664][1:0x7fc5224cc700], file:index-9--3195476868760592993.wt, WT_SESSION.open_cursor: __posix_open_file, 808: /data/db/index-9--3195476868760592993.wt: handle-open: open: Operation not permitted"}}
{"t":{"$date":"2022-09-13T18:39:51.104+00:00"},"s":"F",  "c":"STORAGE",  "id":50882,   "ctx":"AuthorizationManager-1","msg":"Failed to open WiredTiger cursor. This may be due to data corruption","attr":{"uri":"table:index-9--3195476868760592993","config":"overwrite=false","error":{"code":8,"codeName":"UnknownError","errmsg":"1: Operation not permitted"},"message":"Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair"}}
{"t":{"$date":"2022-09-13T18:39:51.104+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"AuthorizationManager-1","msg":"Fatal assertion","attr":{"msgid":50882,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp","line":109}}
{"t":{"$date":"2022-09-13T18:39:51.104+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"AuthorizationManager-1","msg":"\n\n***aborting after fassert() failure\n\n"}

So why is there operation is not permitted? I already run mongod --repair, but the error still occurs.

This is how the pod is deployed:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      hostname: mongodb
      # securityContext:
      #   runAsUser: 999
      #   runAsGroup: 3000
      #   fsGroup: 2000
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data
      containers:
        - name: mongodb
          image: mongo:5.0.12
          args: ["--auth", "--dbpath", "/data/db"]
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 27017
          volumeMounts:
            - mountPath: /data/db
              name: data
          # securityContext:
          #   allowPrivilegeEscalation: false

Update

The PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
2 Answers
Related