Cassandra claims "A node with address X already exists" when restarting that same node

user_786947 0
Viewed 124

So I'm running cassandra in kubernetes. Every once in a while, one of them will fall over or run out of disk or.. look. Things happen to people.

This is the status I found in the ring this morning.

root@cassandra-cluster-arm-az-a-3:/# nodetool status
Datacenter: eu-central-1-arm
============================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load       Tokens       Owns (effective)  Host ID                               Rack
DN  192.168.132.73   1.08 TiB   256          28.3%             b583b2c7-c5a4-4f9e-90a9-eddc4f56b8b2  eu-central-1-arm-az-a
UN  192.168.206.106  1.11 TiB   256          29.7%             2f653ab5-b97c-4756-84b5-841eedae9419  eu-central-1-arm-az-b
UN  192.168.179.26   1.15 TiB   256          31.2%             f79153d2-01f9-4687-a4b4-989f0696e68b  eu-central-1-arm-az-a
DN  192.168.182.250  1.1 TiB    256          29.1%             e23ca25b-2a21-467a-b518-16375439b834  eu-central-1-arm-az-a
UN  192.168.179.237  1.09 TiB   256          29.0%             6eb40e00-d829-491d-a1a8-c3668b3fa6d7  eu-central-1-arm-az-a
UN  192.168.228.174  1.17 TiB   256          32.6%             cbdd095d-502f-4720-a937-8314200dd8f1  eu-central-1-arm-az-b
UN  192.168.247.177  1.24 TiB   256          33.0%             0d0fc344-aaf9-4cb7-9298-2025e89c672b  eu-central-1-arm-az-b
UN  192.168.234.100  1.12 TiB   256          30.0%             3971e56c-cde3-4507-bc40-2dbe5c9ff549  eu-central-1-arm-az-b
UN  192.168.147.148  1.1 TiB    256          29.2%             abfe9961-c737-4d8f-935e-1043111cd170  eu-central-1-arm-az-a
UN  192.168.193.117  1.07 TiB   256          27.9%             85b5d818-fc9c-48cb-ac82-2e8f2d82c9b3  eu-central-1-arm-az-b

I log onto 192.168.182.250 and run nodetool status on it. Yeah. It's stuck and can't contact any of the other nodes. I kill the process and kubernetes restarts the node. Upon startup, I end up in a crashloop of these:

ERROR [main] 2022-05-17 07:01:56,176 CassandraDaemon.java:708 - Exception encountered during startup
java.lang.RuntimeException: A node with address /192.168.182.250 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:558) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:807) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:667) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:379) [apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602) [apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [apache-cassandra-3.11.2.jar:3.11.2]
INFO [StorageServiceShutdownHook] 2022-05-17 07:01:56,180 HintsService.java:220 - Paused hints dispatch
WARN [StorageServiceShutdownHook] 2022-05-17 07:01:56,183 Gossiper.java:1548 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
INFO [StorageServiceShutdownHook] 2022-05-17 07:01:56,183 MessagingService.java:984 - Waiting for messaging service to quiesce
INFO [ACCEPT-/192.168.182.250] 2022-05-17 07:01:56,185 MessagingService.java:1338 - MessagingService has terminated the accept() thread
INFO [StorageServiceShutdownHook] 2022-05-17 07:01:56,574 HintsService.java:220 - Paused hints dispatch
192.168.182.250
Starting Cassandra on 192.168.182.250

The node, having IP 192.168.182.250, cannot join because "A node with address /192.168.182.250 already exists".

.. Why though?

Note. These Cassandras are running on an AWS, ARM nodes, with persistent volumes attached.

Full(-ish) startup log https://gist.github.com/zoomix/2e95b9350da97a2e1f5bc1913045826a

cassandra
1 Answers

There's a good chance that in your Kubernetes setup that you're adding a pod with the same IP address but not reusing the persistent volume so there's no data/ directories on it with data from the pod which failed/ran out of disk.

When this happens, the cluster thinks you're adding a new node with the same IP and it won't allow you because another node in the cluster with the same IP already exists.

Our recommendation is to use persistent volumes (PV) so that when replacing a failed pod, the same PV gets mounted to the new pod and thus the node's local metadata (data in the node's system.local table) is preserved. If Cassandra is able to check the contents of the system.local table on startup, then it knows that it is the same node re-joining the cluster.

In case you weren't already aware, we've created K8ssandra.io -- our opinionated deployment of Apache Cassandra on Kubernetes (fully open-source and free) using the cass-operator and with all the admin tools already bundled in:

  • Reaper for automated repairs
  • Medusa for backups and restores
  • Metrics Collector for monitoring with Prometheus + Grafana
  • Traefik templates for k8s cluster ingress
  • Stargate.io - a data gateway for connecting to Cassandra using REST API, GraphQL API, JSON/Doc API and gRPC API

Cheers!

user_42695350
Related