Why do my Cassandra hints files keep getting corrupted

Viewed 2485

We're running multiple Kubernetes clusters, which run Cassandra. Our usual procedure when doing a rolling restart of the Cassandra pods is to log into each and submit a nodetool drain and then trigger a recreation of that pod. But often when the pods restart we get errors like

ERROR [HintsDispatcher:2] 2017-08-07 11:09:32,489 HintsDispatchExecutor.java:243 - Failed to dispatch hints file 5fdd139d-4465-4825-85ef-f380bddcb67d-1502100535128-1.hints: file is corrupted ({})

Those corrupt files prevent Cassandra from starting. Is there a way to tell Cassandra to flush all buffers and stop writing, before stopping it, to ensure there are no corrupt files left behind?

1 Answers
Related