I have a 12 node cluster on AWS on Cassandra 1.2.11 (DSE). I lost one of the nodes because it lost the ephemeral drive on Amazon (which contained the data). To deal with this, I removed the node with nodetool removenode $hostid which worked. The cluster still appears to be balanced, etc.
The problem is that when I tried to bootstrap a new node, I am now getting errors like this:
java.lang.IllegalStateException: unable to find sufficient sources for streaming range (-2556758013916855401,-2545694469859252228]
at org.apache.cassandra.dht.RangeStreamer.getRangeFetchMap(RangeStreamer.java:205)
at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:129)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:975)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:741)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:585)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:482)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:351)
at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:381)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)
Cannot load daemon
Service exit with a return value of 3
What doesn't make sense is that other than the Opscenter which is using SimpleStrategy, the rest of the keyspaces are all using NetworkTopologyStrategy and have an RF of 3. The way I have been attempting to deal with this is figuring out which node has the range that is failing on and running a nodetool repair -pr and then trying the bootstrap again. While this might eventually work once I run a repair around the entire cluster (which could take days), I am down a node and the cluster is running in a degraded state. And if I lose another node, I am sort of screwed.
What should I be doing here and how can I get around this issue and force the node to bootstrap?