Rabbitmq containers throwing error "Bad characters in cookie"

Viewed 162

I am trying to setup a Rabbitmq cluster and when the containers start they are failing with error [error] CRASH REPORT Process <0.200.0> with 0 neighbours crashed with reason: "Bad characters in cookie" in auth:init_no_setcookie/0 line 313. This suggests that the erlang cookie value passed in is not valid :

kubectl -n demos get pods
NAME                                  READY   STATUS             RESTARTS   AGE
mongodb-deployment-6499999-vpcjh   1/1     Running            0          12h
rabbitmq-0                            0/1     CrashLoopBackOff   9          25m
rabbitmq-1                            0/1     CrashLoopBackOff   9          24m
rabbitmq-2                            0/1     CrashLoopBackOff   9          23m

And when I query the logs for one of the pods :

kubectl -n demos logs -p rabbitmq-0 --previous

I get :

WARNING: '/var/lib/rabbitmq/.erlang.cookie' was populated from 

'$RABBITMQ_ERLANG_COOKIE', which will no longer happen in 3.9 and later! (https://github.com/docker-library/rabbitmq/pull/424)
Configuring logger redirection
02:04:47.506 [error] Bad characters in cookie
02:04:47.512 [error]
02:04:47.506 [error] Supervisor net_sup had child auth started with auth:start_link() at undefined exit with reason "Bad characters in cookie" in auth:init_no_setcookie/0 line 313 in context start_error

02:04:47.506 [error] CRASH REPORT Process <0.200.0> with 0 neighbours crashed with reason: "Bad characters in cookie" in auth:init_no_setcookie/0 line 313
02:04:47.522 [error] BOOT FAILED
BOOT FAILED
02:04:47.523 [error] ===========
===========
02:04:47.523 [error] Exception during startup:
Exception during startup:
02:04:47.524 [error]

02:04:47.524 [error]     supervisor:children_map/4 line 1250
....
....
....

This is how I am generating the cookie in bash :

dd if=/dev/urandom bs=30 count=1 | base64

And in the secrets manifest I have :

metadata:
name: rabbit-secret
  namespace: demos
type: Opaque
data:
  # echo -n "cookie-value" | base64
  RABBITMQ_ERLANG_COOKIE: <encoded_cookie_value_here>

And in statefulset I have :

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: demos
spec:
  serviceName: rabbitmq
  replicas: 3
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      serviceAccountName: rabbitmq
      initContainers:
      - name: config
        image: busybox
        imagePullPolicy: "IfNotPresent"
        command: ['/bin/sh', '-c', 'cp /tmp/config/rabbitmq.conf /config/rabbitmq.conf && ls -l /config/ && cp /tmp/config/enabled_plugins /etc/rabbitmq/enabled_plugins']
        volumeMounts:
        - name: config
          mountPath: /tmp/config/
          readOnly: false
        - name: config-file
          mountPath: /config/
        - name: plugins-file
          mountPath: /etc/rabbitmq/
      containers:
      - name: rabbitmq
        image: rabbitmq:3.8-management
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 4369
          name: discovery
        - containerPort: 5672
          name: amqp
        env:
        - name: RABBIT_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: RABBIT_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: RABBITMQ_NODENAME
          value: rabbit@$(RABBIT_POD_NAME).rabbitmq.$(RABBIT_POD_NAMESPACE).svc.cluster.local
        - name: RABBITMQ_USE_LONGNAME 
          value: "true"
        - name: RABBITMQ_CONFIG_FILE
          value: "/config/rabbitmq"
        - name: RABBITMQ_ERLANG_COOKIE
          valueFrom:
            secretKeyRef:
              name: rabbit-secret
              key: RABBITMQ_ERLANG_COOKIE
        - name: K8S_HOSTNAME_SUFFIX
          value: .rabbitmq.$(RABBIT_POD_NAMESPACE).svc.cluster.local
        volumeMounts:
        - name: data
          mountPath: /var/lib/rabbitmq
          readOnly: false
        - name: config-file
          mountPath: /config/
        - name: plugins-file
          mountPath: /etc/rabbitmq/
      volumes:
      - name: config-file
        emptyDir: {}
      - name: plugins-file
        emptyDir: {}
      - name: config
        configMap:
          name: rabbitmq-config
          defaultMode: 0755
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "cinder-csi"
      resources:
        requests:
          storage: 50Mi
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  namespace: labs
spec:
  clusterIP: None
  ports:
  - port: 4369
    targetPort: 4369
    name: discovery
  - port: 5672
    targetPort: 5672
    name: amqp
  selector:
    app: rabbitmq

What am I missing ?

Is there a recommended way of generating the cookie or something else to do with the K8s cluster itself.

I have followed the example given here with the only difference being that I am generating my cookie on my local machine and not the k8s host.

1 Answers

duplicate question?

This requires you to create the Secret. Just to bring this up, you can run a one-off imperative command to create a random Secret:

kubectl create secret generic rabbitmq \
  --from-literal=erlangCookie=$(dd if=/dev/urandom bs=30 count=1 | base64)

The error is from the rabbitmq's source code file docker-entrypoint.sh

if [ "${RABBITMQ_ERLANG_COOKIE:-}" ]; then
    cookieFile='/var/lib/rabbitmq/.erlang.cookie'
    if [ -e "$cookieFile" ]; then
        if [ "$(cat "$cookieFile" 2>/dev/null)" != "$RABBITMQ_ERLANG_COOKIE" ]; then
            echo >&2
            echo >&2 "warning: $cookieFile contents do not match RABBITMQ_ERLANG_COOKIE"
            echo >&2
        fi
    else
        echo "$RABBITMQ_ERLANG_COOKIE" > "$cookieFile"
    fi
    chmod 600 "$cookieFile"
fi

Therefore you can delete '/var/lib/rabbitmq/.erlang.cookie' file, the code will copy the content from environment variable $RABBITMQ_ERLANG_COOKIE to create this file.

If you are working for the production system, you should be very careful and test it in your local system firstly and gain experience.

This source code will be execuated only the rabbitmq restart, and won't run again after it is execuated. And you can find the actual rabbitmq's cookie data from the erlang's console by erlang:get_cookie() -> Cookie | nocookie.

Related