Getting the same errors at the exact time one hour apart, how can I fix them

user_1577343 0
Viewed 44

I am getting this two errors Validator.java:268 - Failed creating a merkle tree for and CassandraDaemon.java:228 - Exception in thread Thread at the exact time 0t:00:03 each hour

cat debug.log | grep -i java | grep -i err | grep -i -oE '.{0,80}ValidationExecutor.{0,85}'
ERROR [ValidationExecutor:6067] 2021-12-10 06:00:09,358 Validator.java:268 - Failed creating a merkle tree for [repair #6a9c43e0-597e-11ec-954f-0be7ea52f380 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6067] 2021-12-10 06:00:09,359 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6067,1,main]
ERROR [ValidationExecutor:6068] 2021-12-10 06:00:15,555 Validator.java:268 - Failed creating a merkle tree for [repair #6ac29090-597e-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6068] 2021-12-10 06:00:15,556 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6068,1,main]
ERROR [ValidationExecutor:6070] 2021-12-10 07:00:03,236 Validator.java:268 - Failed creating a merkle tree for [repair #cc2b9220-5986-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6071] 2021-12-10 07:00:03,236 Validator.java:268 - Failed creating a merkle tree for [repair #cc2c2e60-5986-11ec-954f-0be7ea52f380 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6070] 2021-12-10 07:00:03,237 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6070,1,main]
ERROR [ValidationExecutor:6071] 2021-12-10 07:00:03,237 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6071,1,main]
ERROR [ValidationExecutor:6073] 2021-12-10 08:00:03,422 Validator.java:268 - Failed creating a merkle tree for [repair #2e0ad520-598f-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6073] 2021-12-10 08:00:03,423 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6073,1,main]
ERROR [ValidationExecutor:6074] 2021-12-10 08:00:03,424 Validator.java:268 - Failed creating a merkle tree for [repair #2e1a8c90-598f-11ec-a28d-115f730d3c5f on mars/mapping, [(372614885
ERROR [ValidationExecutor:6074] 2021-12-10 08:00:03,425 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6074,1,main]
ERROR [ValidationExecutor:6076] 2021-12-10 09:00:03,566 Validator.java:268 - Failed creating a merkle tree for [repair #8fe906b0-5997-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6076] 2021-12-10 09:00:03,569 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6076,1,main]
ERROR [ValidationExecutor:6077] 2021-12-10 09:00:03,655 Validator.java:268 - Failed creating a merkle tree for [repair #8ff032a0-5997-11ec-a28d-115f730d3c5f on mars/mapping, [(372614885
ERROR [ValidationExecutor:6077] 2021-12-10 09:00:03,656 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6077,1,main]
ERROR [ValidationExecutor:6079] 2021-12-10 10:00:03,833 Validator.java:268 - Failed creating a merkle tree for [repair #f1bf9720-599f-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6080] 2021-12-10 10:00:03,833 Validator.java:268 - Failed creating a merkle tree for [repair #f1cbf330-599f-11ec-954f-0be7ea52f380 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6079] 2021-12-10 10:00:03,834 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6079,1,main]
ERROR [ValidationExecutor:6080] 2021-12-10 10:00:03,834 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6080,1,main]
ERROR [ValidationExecutor:6083] 2021-12-10 11:00:20,044 Validator.java:268 - Failed creating a merkle tree for [repair #53a0d5f0-59a8-11ec-a28d-115f730d3c5f on mars/mapping, [(372614885
ERROR [ValidationExecutor:6082] 2021-12-10 11:00:20,044 Validator.java:268 - Failed creating a merkle tree for [repair #538d9c10-59a8-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6083] 2021-12-10 11:00:20,045 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6083,1,main]
ERROR [ValidationExecutor:6082] 2021-12-10 11:00:20,045 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6082,1,main]
ERROR [ValidationExecutor:6085] 2021-12-10 12:00:03,054 Validator.java:268 - Failed creating a merkle tree for [repair #b4ec3d60-59b0-11ec-954f-0be7ea52f380 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6085] 2021-12-10 12:00:03,055 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6085,1,main]
ERROR [ValidationExecutor:6085] 2021-12-10 12:00:03,563 ActiveRepairService.java:561 - Cannot start multiple repair sessions over the same sstables
ERROR [ValidationExecutor:6085] 2021-12-10 12:00:03,563 Validator.java:268 - Failed creating a merkle tree for [repair #b533a6a0-59b0-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6085] 2021-12-10 12:00:03,564 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6085,1,main]

What is the cause of them and the recurrence and how to fix them?

cassandra
1 Answers

The logs show that Cassandra failed in the validation phase of the anti-entropy repair process.

As "Cannot start multiple repair sessions over the same sstables" this means there're multiple repair sessions on the same token range at the same time.

You need to make sure that you have no repair session currently running on your cluster, and no anti-compaction.

I suggest rolling restart in order to stop all running repairs. then try to repair node by node.

One last suggestion is to try https://github.com/thelastpickle/cassandra-reaper it used to run automated repairs for Cassandra.

user_19508460
Related