I am trying to setup a Rabbitmq cluster and when the containers start they are failing with error [error] CRASH REPORT Process <0.200.0> with 0 neighbours crashed with reason: "Bad characters in cookie" in auth:init_no_setcookie/0 line 313. This suggests that the erlang cookie value passed in is not valid :
kubectl -n demos get pods
NAME READY STATUS RESTARTS AGE
mongodb-deployment-6499999-vpcjh 1/1 Running 0 12h
rabbitmq-0 0/1 CrashLoopBackOff 9 25m
rabbitmq-1 0/1 CrashLoopBackOff 9 24m
rabbitmq-2 0/1 CrashLoopBackOff 9 23m
And when I query the logs for one of the pods :
kubectl -n demos logs -p rabbitmq-0 --previous
I get :
WARNING: '/var/lib/rabbitmq/.erlang.cookie' was populated from
'$RABBITMQ_ERLANG_COOKIE', which will no longer happen in 3.9 and later! (https://github.com/docker-library/rabbitmq/pull/424)
Configuring logger redirection
02:04:47.506 [error] Bad characters in cookie
02:04:47.512 [error]
02:04:47.506 [error] Supervisor net_sup had child auth started with auth:start_link() at undefined exit with reason "Bad characters in cookie" in auth:init_no_setcookie/0 line 313 in context start_error
02:04:47.506 [error] CRASH REPORT Process <0.200.0> with 0 neighbours crashed with reason: "Bad characters in cookie" in auth:init_no_setcookie/0 line 313
02:04:47.522 [error] BOOT FAILED
BOOT FAILED
02:04:47.523 [error] ===========
===========
02:04:47.523 [error] Exception during startup:
Exception during startup:
02:04:47.524 [error]
02:04:47.524 [error] supervisor:children_map/4 line 1250
....
....
....
This is how I am generating the cookie in bash :
dd if=/dev/urandom bs=30 count=1 | base64
And in the secrets manifest I have :
metadata:
name: rabbit-secret
namespace: demos
type: Opaque
data:
# echo -n "cookie-value" | base64
RABBITMQ_ERLANG_COOKIE: <encoded_cookie_value_here>
And in statefulset I have :
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: rabbitmq
namespace: demos
spec:
serviceName: rabbitmq
replicas: 3
selector:
matchLabels:
app: rabbitmq
template:
metadata:
labels:
app: rabbitmq
spec:
serviceAccountName: rabbitmq
initContainers:
- name: config
image: busybox
imagePullPolicy: "IfNotPresent"
command: ['/bin/sh', '-c', 'cp /tmp/config/rabbitmq.conf /config/rabbitmq.conf && ls -l /config/ && cp /tmp/config/enabled_plugins /etc/rabbitmq/enabled_plugins']
volumeMounts:
- name: config
mountPath: /tmp/config/
readOnly: false
- name: config-file
mountPath: /config/
- name: plugins-file
mountPath: /etc/rabbitmq/
containers:
- name: rabbitmq
image: rabbitmq:3.8-management
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 4369
name: discovery
- containerPort: 5672
name: amqp
env:
- name: RABBIT_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: RABBIT_POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: RABBITMQ_NODENAME
value: rabbit@$(RABBIT_POD_NAME).rabbitmq.$(RABBIT_POD_NAMESPACE).svc.cluster.local
- name: RABBITMQ_USE_LONGNAME
value: "true"
- name: RABBITMQ_CONFIG_FILE
value: "/config/rabbitmq"
- name: RABBITMQ_ERLANG_COOKIE
valueFrom:
secretKeyRef:
name: rabbit-secret
key: RABBITMQ_ERLANG_COOKIE
- name: K8S_HOSTNAME_SUFFIX
value: .rabbitmq.$(RABBIT_POD_NAMESPACE).svc.cluster.local
volumeMounts:
- name: data
mountPath: /var/lib/rabbitmq
readOnly: false
- name: config-file
mountPath: /config/
- name: plugins-file
mountPath: /etc/rabbitmq/
volumes:
- name: config-file
emptyDir: {}
- name: plugins-file
emptyDir: {}
- name: config
configMap:
name: rabbitmq-config
defaultMode: 0755
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "cinder-csi"
resources:
requests:
storage: 50Mi
---
apiVersion: v1
kind: Service
metadata:
name: rabbitmq
namespace: labs
spec:
clusterIP: None
ports:
- port: 4369
targetPort: 4369
name: discovery
- port: 5672
targetPort: 5672
name: amqp
selector:
app: rabbitmq
What am I missing ?
Is there a recommended way of generating the cookie or something else to do with the K8s cluster itself.
I have followed the example given here with the only difference being that I am generating my cookie on my local machine and not the k8s host.