Can we add a new Cassandra 3.11 DC, upgrade it to 4.0, then eventually get rid of the old DC?

user_5539664 0
Viewed 269

We want to upgrade from cassandra 3.11.12 to cassandra 4.0.2 using multi DC replication. We want to do that for easy and quick rollback (avoid a situation of backup\snap and restore).

Steps we think to do:
1. lock app to use (read & writes) only with old\current dc (using driver options).
3. create new dc with same version (3.11.12) - new dc will created with 16 num_tokens (today we are with 3.11 default 256 and want to move to 16).
3. sync all data\keyspaces to new dc - and keep the sync active to the 2 DC's.
4. upgrade the new dc from 3.11.12 to 4.0.2
5. after step 4 done, move app to use only new DC (version 4 after upgrade)
6. wait few days to see all going well
7. stop replication to old dc
8. remove old dc nodes until and stay only with the new DC (cassandra 4.0.2)

A. The main question is this process should work?
B. Is there a problem with move to 16 num_tokens this way?
C. Is it ok to keep sync the data\keyspaces between the 2 DCs for a few days while they are in different versions of Cassandra (dc1 in 3.11.12 , dc2 in 4.0.2)?

Note that I have seen that it is not recommended to have a cluster with a mix of cassandra versions but this is only for an upgrade process with a quick and simple rollback. The old DC with the old cassandra version will be removed after a few days when everything seems to work properly with the new version.

cassandra
1 Answers

It is a valid upgrade path but bear in mind that there could be some disadvantages with your proposed approach. For example, if a node goes down (say for a hardware failure) then you won't be able to decommission it.

Any operation that requires streaming will not work in a mixed-version cluster. Those operations include bootstrap, decommission, repairs.

To answer your questions directly:

  • A. Yes, it will work but with some gotchas.
  • B. No, adding a new DC is the only way you can change the number of tokens.
  • C. Yes, replication is designed to work in mixed-versions.

To answer the question you didn't ask: rolling back an upgrade is actually quite rare in my experience. You would typically upgrade one node at a time. If you run into a problem on a node, you would fix that node then proceed with the rolling upgrade until all nodes in the cluster have been upgraded.

During the rolling upgrade, your application should continue to work and so there should be no reason to perform a rollback. Cheers!

user_42695350
Related