Inconsistent Cassandra system.peers state

Question

We are running 6 nodes Cassandra (3.11.2) cluster on Kubernetess. Recently I noticed that the data in system.peers table is inconsistent. However, data in system.local seems to be ok. nodetool describecluster also doesn't report any issues.

Below you will find the anonymized result of system.peers and system.local queries. I executed them by forwarding port to a single node at a time (I hope this allows skipping load balancing policy and accessing node directly)

Is that state of a system.peers table is harmful? Or maybe it is expected?

SELECT peer, schema_version FROM system.peers

node 0
peer | schema_version
IP1 | schema2
IP2 | schema1
IP3 | schema1
IP4 | null
IP5 | schema1
IP6 | schema1
IP7 | schema1

node 1
peer | schema_version
IP8 | null
IP9 | schema1
IP3 | schema1
IP5 | schema1
IP6 | schema1
IP7 | schema1

node 2
peer | schema_version
IP11 | null
IP2 | schema1
IP9 | schema1
IP3 | schema1
IP4 | schema3
IP10 | null
IP5 | schema1
IP6 | schema1

node 3
peer | schema_version
IP12 | schema3
IP2 | schema1
IP9 | schema1
IP13 | null
IP3 | schema1
IP5 | schema1
IP7 | schema1

node 4
peer | schema_version
IP2 | schema1
IP9 | schema1
IP3 | schema1
IP6 | schema1
IP7 | schema1

node 5
peer | schema_version
IP8 | schema3
IP2 | schema1
IP9 | schema1
IP5 | schema1
IP6 | schema1
IP7 | schema1

SELECT key, broadcast_address, schema_version FROM system.local

node 0
key | broadcast_address | schema_version
local | IP9 | schema1

node 1
key | broadcast_address | schema_version
local | IP2 | schema1

node 2
key | broadcast_address | schema_version
local | IP7 | schema1

node 3
key | broadcast_address | schema_version
local | IP6 | schema1

node 4
key | broadcast_address | schema_version
local | IP5 | schema1

node 5
key | broadcast_address | schema_version
local | IP3 | schema1

nodetool describecluster

Cluster Information:
  Name: CLUSTER_NAME
  Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch
  DynamicEndPointSnitch: enabled
  Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
  Schema versions:
    e718e690-d474-376e-8020-ed0eba5b6797: [IP5, IP9, IP3, IP2, IP6, IP7]

user_4406006 · Answer

This is unexpected, but has been known to happen, for example: CASSANDRA-7122, CASSANDRA-7531.

This might cause issues for different client drivers (for example, see JAVA-852 and JAVA-2280), although most client libraries will ignore corrupt peers records like this and log a warning when it happens.

Since you mention Kubernetes, is it possible you are replacing nodes frequently? I wonder if there is a latent bug in C* where it isn't properly removing old peers entries. There have been issues reported in the past that have been closed with COULD NOT REPRODUCE.

If you can reproduce this rather easily, it would be super helpful to the community if you could create a JIRA ticket describing the issue and how to reproduce it. Otherwise if you don't have the time, if you could describe your kubernetes set up (i.e. are you using a community operator or something else?) and explain some of the operations you may be doing that could contribute to this (i.e. replacing nodes) I could look into it when I get some time.

Inconsistent Cassandra system.peers state

1 Answers