Cassandra: Shipping Disk to New DC in order to sync 50TB of data

user_1637506 0
Viewed 131

We're adding a new datacenter to our Cassandra cluster. Currently, we have a 15-node DC with RF=3 resulting in about 50TB~ of data.

We are adding another datacenter in a different country and we want both data centers to contain all the data. Obviously, synchronizing 50TB of data across the internet will take a gargantuan amount of time.

Is it possible to copy a full back to a few disks, ship that to the new DC and then recover? I'm just wondering what would be the procedure to do so.

Could someone give me a few pointers on this operation, if possible at all? Or any other tips?

Our new DC is going to be smaller (6 nodes) for the time being, although enough space will be available. The new DC is mostly meant as a live-backup/failover and will not be the primary cluster for writing, generally speaking.

cassandra-3.0 database-replication replication cassandra database
1 Answers

TL;DR; Due to the topology (node count) change between the two DCs, avoiding streaming the data in isn't possible AFAIK.

Our new DC is going to be smaller (6 nodes) for the time being

The typical process isn't going to work due to token alignment on the nodes being different (new cluster's ring will change). So just copying the existing SSTables wont work, as the nodes that hold those tables, might not have the tokens corresponding to the data in the files and so C* wont be able to find said data.

Bulk loading the data to the new DC is out too, as you'll be overwriting the old data if you re-insert it.

To give you an overview of the process if you were to retain the topology:

  1. snapshot the data from the original DC
  2. Configure the new DC. It's extremely important that you set initial_token for each machine. You can get a list of what tokens you need by running nodetool ring on the original cluster. This is why you need the same number of nodes. As importantly, when copying the SSTable files over, you need the files and the tokens to be from the same node.
  3. ship the data to the new DC (Remember if the new node 10.0.0.1 got it's tokens from 192.168.0.100 in the old dc, then it also has to get it's snapshot data from 192.168.0.100).
  4. Start the new DC and ensure both DCs see eachother ok.
  5. Rebuild and repair system_distributed and system_auth (assuming you have authentication enabled)
  6. Update client consistency to whatever you need. (Do you want to write to both DCs? From your description sounds like a no so you might be all good).
  7. Update the schema, ensure that you're using NetworkTopologyStrategy for any keyspce that you want to be shared, then add some replication for the new DC.
    ALTER KEYSPACE ks WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', 'oldDC' : 3, 'newDC':3 };
  1. Run a full repair on each node in the new dc.
user_11594720
Related