How to cluster embedded Cassandra instances?

Viewed 2294

I've written a small application to start an embedded instance of Cassandra 1.2.

I'm trying to create a cluster of 3 of these embedded instances locally, by running 3 instances of this application. Each one looks at a different cassandra.yaml on the filesystem. Each file has:

  • the same cluster name
  • blank initial_token
  • unique listen address (all mapped to 127.0.0.1 in my hosts file)
  • unique rpc, storage and ssl_storage ports
  • the same seed (the listen address (no port) of the first server)
  • unique -Dcom.sun.management.jmxremote.port value passed in application launch

When I launch the applications, all come up fine, and have separate storage on the filesystem. However, when I use nodetool to inspect each one, each appears to be in a cluster by itself:

C:\Program Files\DataStax Community\apache-cassandra\bin>nodetool -h 127.0.0.1 -p 7197 ring
Starting NodeTool

Datacenter: datacenter1
==========
Replicas: 1

Address    Rack        Status State   Load            Owns                Token

127.0.0.1  rack1       Up     Normal  198,15 KB       100,00%             8219116491729144532


C:\Program Files\DataStax Community\apache-cassandra\bin>nodetool -h 127.0.0.2 -p 7198 ring
Starting NodeTool

Datacenter: datacenter1
==========
Replicas: 1

Address    Rack        Status State   Load            Owns                Token

127.0.0.2  rack1       Up     Normal  152,13 KB       100,00%             -3632227916915216562

Blogs and docs online suggest this should be sufficient. Is it possible to cluster embedded instances? If so, does anyone know how my configuration or understanding is incorrect/insufficient?

Code to launch the embedded instances is below. Hope you can help, thanks.

public class EmbeddedCassandraDemo {

    private static final String CONF_PATH_FORMAT = "D:\\embedded_cassandra\\Node%d\\";

    private ExecutorService executor = Executors.newSingleThreadExecutor();
    private CassandraDaemon cassandraDaemon;
    private int nodeNumber;

    public EmbeddedCassandraDemo(int nodeNumber) {
        this.nodeNumber = nodeNumber;
    }

    public static void main(String [ ] args) throws InterruptedException, ConnectionException {
        new EmbeddedCassandraDemo(Integer.parseInt(args[0])).run();
    }

    private void run() throws InterruptedException, ConnectionException {
        setProperties();

        activateDeamon();
    }

    private void activateDeamon() {
        executor.execute( new Runnable(){

            @Override
            public void run() {
                cassandraDaemon = new CassandraDaemon();
                cassandraDaemon.activate();
            }});
    }

    private void setProperties() {
        System.setProperty("cassandra.config", String.format("file:%scassandra.yaml", String.format(CONF_PATH_FORMAT, nodeNumber)));
        System.setProperty("log4j.configuration", String.format("file:%slog4j-server.properties", String.format(CONF_PATH_FORMAT, nodeNumber)));
        System.setProperty("cassandra-foreground", "true");
    }
}
2 Answers

"blank initial_token"

Are you using virtual nodes? If not, I wonder if that could be your issue. You should have each machine defined with a different initial token. For a 3-node cluster, those initial tokens should be increments of 56,713,727,820,156,410,577,229,101,238,628,035,242 apart from each other.

Using DataStax's Python script for computing initial tokens, these values should suit your needs:

node 0: 0
node 1: 56713727820156410577229101238628035242
node 2: 113427455640312821154458202477256070485

Also, which endpoint_snitch are you using? If you are using "PropertyFileSnitch" make sure that your cassandra-topology.properties file contains a definition for each node (along with DC and rack).

Give that a try and see if it helps.

Related