I've got three cassandra datacenters, with a keyspace that looks like this:
CREATE KEYSPACE allotments WITH replication = {'class': 'NetworkTopologyStrategy', 'hk1': '3', 'hk2': '3', 'hk3': '3'} AND durable_writes = false;
I installed the latest clusters, HK2 and HK3 recently and had auto_bootstrap set to true (by commenting the line).
I then proceeded to run the rebuild command on all of the nodes in the new datacenters (HK2 and HK3) using the original datacenter (HK1) as the source:
nodetool rebuild -- hk1
However, the problem I find myself with is that HK2 seems to be missing data and I am unable to figure out why or fix the problem.
On HK1 (correct data):
cqlsh> use allotments;
cqlsh:allotments> select count(*) from freesale_allotments;
count
-------
22035
(1 rows)
Warnings :
Aggregation query used without partition key
cqlsh:allotments>
On HK2 (missing data):
cqlsh:allotments> use allotments;
cqlsh:allotments> select count(*) from freesale_allotments;
count
-------
19142
(1 rows)
Warnings :
Aggregation query used without partition key
Toubleshooting steps tried so-far:
I tried to rebuild the node again, resetting local data. It would just return within 2-3 seconds:
nodetool rebuild -m reset -- hk1
I tried running a repair on the keyspace across all nodes, this completed successfully after ~1 hour but the count remains unchanged.
nodetool repair allotments -pr
I have tried running the scrub command to clean all of the tables of corrupted data, and again - no change.
nodetool scrub
Can somebody help point me in the right direction to finding out how it has happened, and how i would go about fixing it.
Thanks very much
Martin