Way to determine healthy Cassandra cluster?

user_8971949 0
Viewed 8103

I've been tasked with re-writing some sub-par Ansible playbooks to stand up a Cassandra cluster in CentOS. Quite frankly, there doesn't seem to be much information on Cassandra out there.

I've managed to get the service running on all three nodes at the same time, using the following configuration file, info scrubbed.

HOSTIP=10.0.0.1
MSIP=10.10.10.10
ADMIN_EMAIL=my@email.com
LICENSE_FILE=/tmp/license.conf
USE_LDAP_REMOTE_HOST=n

ENABLE_AX=y
MP_POD=gateway

REGION=test-1

USE_ZK_CLUSTER=y
ZK_HOSTS="10.0.0.1 10.0.0.2 10.0.0.3"
ZK_CLIENT_HOSTS="10.0.0.1 10.0.0.2 10.0.0.3"

USE_CASS_CLUSTER=y
CASS_HOSTS="10.0.0.1:1,1 10.0.0.2:1,1 10.0.0.3:1,1"
CASS_USERNAME=test
CASS_PASSWORD=test

The HOSTIP changes depending on which node the configuration file is on.

The problem is, when I run nodetool ring, each node says there's only two nodes in the cluster: itself and one other, seemingly random from the other two.

What are some basic sanity checks to determine a "healthy" Cassandra cluster? Why is nodetool saying each one thinks there's a different node missing from the cluster?

centos cassandra
3 Answers

nodetool status - overview of the cluster (load, state, ownership)

nodetool info - more granular details at the node-level

As for the node mismatch I would check the following: 

cassandra-topology.properties - identical across the cluster (all 3 IPs listed)

cassandra.yaml - I typically keep this file the same across all nodes. The parameters that MUST stay the same across the cluster are: cluster_name, seeds, partitioner, snitch).

verify all nodes can reach each other (ping, telnet, etc)

DataStax (Cassandra Vendor) has some good documentation. Please note that some features are only available on DataStax Enterprise -

http://docs.datastax.com/en/landing_page/doc/landing_page/current.html

Also check out the Apache Cassandra site -

http://cassandra.apache.org/community/

As well as the user forums -

https://www.mail-archive.com/user@cassandra.apache.org/

user_51371100

Actually, the thing you really want to check is if all the nodes "AGREE" on schema_id. nodetool status shows if nodes or up, down, joining, yet it does not really mean 'healthy' enough to make schema changes or do other changes. The simplest way is: nodetool describecluster

Cluster Information:
        Name: FooBarCluster
        Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch
        DynamicEndPointSnitch: enabled
        Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
        Schema versions:
                43fe9177-382c-327e-904a-c8353a9df590: [10.136.2.1, 10.136.2.2, 10.136.2.3]

If schema IDs do not match, you need to wait for schema to settle, or run repairs, say for example like this:

43fe9177-382c-327e-904a-c8353a9df590: [10.136.2.1, 10.136.2.2]
43fe9177-382c-327e-904a-c8353a9dxxxx: [10.136.2.3]

However, running nodetool is 'heavy' and hard to parse.

The information is inside the database, you can check here:

'SELECT schema_version, release_version FROM  system.local' and
'SELECT peer, schema_version, release_version FROM system.peers'

Then you compare schema_version across all nodes... if they match, the cluster is very likely healthy. You should ALWAYS check this before making any changes to schema.

Now, during a rolling upgrade, when changing engine versions, the release_version is different, so to support automatic rolling upgrades, you need to check schema_id matching within release_versions separately.

user_130669700

I'm not sure all of the problems you might be having, but...

Check the cassandra.yaml file. You need minimum 3 things to be the same - seeds: list (but do not list all nodes as seeds!), cluster_name, and snitch. Make sure your listen_address is correct.

If you are using gossipingPropertyFileSnitch then check cassandra-topology.properties and/or cassandra-rackdc.properties files for accuracy.

Don't start all the nodes at the same time. Start the seed nodes 1st - the other nodes will "gossip" with the seed node to learn cluster topology. Shutdown the seed nodes last.

Don't use shared storage. That defeats the purpose of distributed data and is considered a cassandra anti-pattern.

If you're in AWS, don't use auto-scaling groups unless you know what you're doing.

Once you've done all that, use nodetool status | ring | info or jmx to see what the cluster is doing.

Datastax does have decent documentation for cassandra.

user_31821390
Related