Our DBA team (me) recently took over our infrastructure of Redis cluster with 8 primary/8 secondary servers along with 20-30 other standalone and Sentinel instances.
As a paranoid DBA, one thing I wanted to do was set up a scheduled time for backups using a simple script I wrote that does a save then moves the resulting rdb file to a new location. Since scheduling this script to run each night at 8pm, we have seen multiple nodes in the cluster fail over to the secondary, which I then have to move back manually. Now, there are also saves going on via the auto save settings so I may just abandon doing explicit backups altogether but I'm curious what's going on. Would doing bgsave instead solve the problem ?
October 13th 2021, 20:00:07.176 * DB saved on disk
October 13th 2021, 20:00:11.179 # Client id=374070 addr=10.100.0.151:48416 fd=25 name= age=14 idle=0 flags=P db=0 sub=15 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=6 omem=101109904 events=rw cmd=subscribe scheduled to be closed ASAP for overcoming of output buffer limits.
October 13th 2021, 20:00:16.181 * FAIL message received from 6c4a3e7c46d31d123dd5b55b662d323d51467109 about 866744a3afd0c83cbb91234313ff6ae3c4a9dfec
October 13th 2021, 20:00:18.182 * Clear FAIL state for node 866744a3afd0c83cbb91234313ff6ae3c4a9dfec: replica is reachable again.
October 13th 2021, 20:00:18.182 * FAIL message received from c4f0c574aaaa1309b6ec514a9645a5f6e3238e34 about 06de5562d113d2c36977cbf581b013487d19ee4e
October 13th 2021, 20:00:20.183 * Clear FAIL state for node 06de5562d113d2c36977cbf581b013487d19ee4e: replica is reachable again.
October 13th 2021, 20:00:20.183 * FAIL message received from b9f16fd70418a297796cb0a5ad6cad312aa6d784 about 8dd6dc1ff3105fbcd49e6c541c605f2b9d364952
October 13th 2021, 20:00:20.183 # Cluster state changed: fail
October 13th 2021, 20:00:21.183 # Cluster state changed: ok