I just set up a three node Galera cluster (10.9.2-MariaDB) last week. Everything went smoothly, I could sync nodes in seconds and bounce at will.
Yesterday I encountered a major problem without any apparent modifications to the system. Now If I restart mariadb on one of the nodes I can see in the logs that it joins the cluster immediately. However the time it takes WSREP to request state transfer from the donor is always 560 seconds (give or take 2 secs). Otherwise, the cluster operates normally (replication is immediate etc.)
Here is a snippet from the logs when the server web-cluster-db2 has been restarted (note the lag time before "Member requested state transfer"). Does anyone have a clue what's going on?
Thanks in advance,
-S
Donor log:
View:
id: ffad2f07-3412-11ed-802e-e3c787820b3f:541
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 1
members(3):
0: ae01cc87-3420-11ed-b896-efa50b185b45, web-cluster-db1
1: c60d47de-3420-11ed-8445-16d58f164eb1, web-cluster-db3
2: cca24476-342d-11ed-9fbf-4f610f46cd7c, web-cluster-db2
=================================================
2022-09-14 13:04:43 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-09-14 13:04:43 2 [Note] WSREP: Lowest cert index boundary for CC from group: 518
2022-09-14 13:04:43 2 [Note] WSREP: Min available from gcache for CC from group: 5
2022-09-14 13:04:46 0 [Note] WSREP: (c60d47de-8445, 'tcp://0.0.0.0:4567') turning message relay requesting off
2022-09-14 13:14:06 0 [Note] WSREP: Member 2.0 (web-cluster-db2) requested state transfer from '*any*'. Selected 1.0 (web-cluster-db3)(SYNCED) as donor.
2022-09-14 13:14:06 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 543)
2022-09-14 13:14:06 2 [Note] WSREP: Detected STR version: 1, req_len: 123, req: STRv1
2022-09-14 13:14:06 2 [Note] WSREP: IST request: ffad2f07-3412-11ed-802e-e3c787820b3f:539-541|tcp://x.x.x.x:4568
2022-09-14 13:14:06 2 [Note] WSREP: Server status change synced -> donor
Joiner log:
2022-09-14 13:04:43 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address 'x.x.x.x' --datadir '/var/lib/mysql/' --parent 14485 --progress 0 --mysqld-args --wsrep_start_position=ffad2f07-3412-11ed-802e-e3c787820b3f:539'
WSREP_SST: [INFO] rsync SST started on joiner (20220914 13:04:44.012)
2022-09-14 13:04:45 0 [Note] WSREP: (cca24476-9fbf, 'tcp://0.0.0.0:4567') turning message relay requesting off
2022-09-14 13:14:06 2 [Note] WSREP: ####### IST uuid:ffad2f07-3412-11ed-802e-e3c787820b3f f: 540, l: 541, STRv: 3
2022-09-14 13:14:06 2 [Note] WSREP: IST receiver addr using tcp://x.x.x.x:4568