Shouldn't hinted handoff in Cassandra prevent deleted data from getting resurrected?

user_1544322 0
Viewed 70

I have a question about the zombie deleted data reappearing in cassandra when we do aggressive compaction and use low gc_grace_seconds.

Based upon the articles that I have read, they say if we get rid of tombstones quickly using lower gc_grace_seconds and other params, lets say if we have a replication factor of 3 and during the tombstone update only 2 of the replicas are up and acknowledge the tombstone. Because of aggressive compaction, those tombstones would be removed along with the shadow data quickly on the two replicas

Now when the one replica which was down before comes backup, it will not be aware of the tombstone and the data being wiped out at the other two replicas. When the read repair happens, this data which wasn't removed on this replica will come back to life and replicated to the other two replicas

But my question is shouldn't hinted handoff take care of it? When the replica comes backup, shouldn't the replica read the hint and fix the data/ delete it at its end. The default expiration period of hinted hand off is 3hrs. So is it the case that it assumes the replica comes back up after the expiry period of hinted hand off or does it consider the fact that the hinted hand off doesn't happen immediately when the replica comes backup. The replica polls every 10 minutes for the hints or through gossip amongst node which will take some time.

cassandra
2 Answers

The assumptions you have made are incorrect and hints alone won't be able to prevent deleted data from getting resurrected. I'll try to clarify some of the misconceptions you have.

  1. You should not "do aggressive compactions". Forcing a major compaction can cause more problems than you are trying to solve as I've explained in this post -- https://community.datastax.com/questions/6396/.

  2. A low GC grace is a band-aid solution. It is only temporarily masking a bigger problem -- you need to address why your table has a lot of tombstones. You should not set GC grace to a low value that's not practical for you to replace a dead node. Let's say a node has a hardware failure and is not recoverable. If it normally takes you 2 days to fix a server, you should set GC grace higher than 2 days.

  3. Coordinators for write requests store hints for replicas which are down. When the replica comes back online, the coordinator replays the hints to the replica. This is what's known as hinted handoff. A downed replica does not "read the hint" itself.

  4. There is no "default expiration" on hints. The 3 hours is the max_hint_window_in_ms which is the amount of time that hints will be stored for a replica that is down. After this window, hints will no longer be stored by the coordinator and you need to repair the replica manually.

  5. Hints older than GC grace are expired and do not get handed off to a replica. This is another danger of setting GC grace too low.

If you need to set GC grace to a low value and manually run compactions, it indicates to me that there's something wrong with your data model to generate so much tombstones. You need to review your data model and address the root cause. Chances are, you are using Cassandra as a queue or storing queue-like datasets which are an anti-pattern since they generate a lot of tombstones as discussed in this blog post.

Ryan Svihla wrote Understanding Deletes in Cassandra where he proposes an alternative model for handling queue-like data that partially avoids the tombstone issue. Cheers!

user_42695350

3 hours is default time for hinted handoff, but if you have gc_grace_seconds lower than that, then hints are expiring before hitting the time set for hinted handoff. There is a good article from the last pickle that explains how these things are related to each other.

Alex_Ott_186270
Related