I'm using redis as a broker for celery in my django project. As part of my deployment process I restart the services at the end, so redis, celery, gunicorn (django) etc starting with redis. However I run into an issue where redis will not shutdown.
$ sudo systemctl restart redis
$
And there it hangs, at time of writing for 15 minutes. journalctl shows no entries (logs have rotated overnight I assume), systemctl status shows the redis unit as deactivating (sig-term) but no indication what it's doing, other than:
May 24 10:31:22 staging systemd[1]: Stopping Advanced key-value store...
May 24 10:31:22 staging run-parts[305]: run-parts: executing /etc/redis/redis-server.pre-down.d/00_example
I understand that sig-term allows redis to exit gracefully, so wondered if the celery beat tasks or the django server was accessing it, but having stopped those services it is still hung up. Are there any places I'm not aware of where I can check the status/what it's doing?
edit: Aha, ok so it turns out redis doesn't actually log to systemd by default, instead it logs to /var/log/redis which does actually yield some information:
31602:M 24 May 10:59:56.097 * 1 changes in 900 seconds. Saving...
31602:M 24 May 10:59:56.101 * Background saving started by pid 1151
1151:C 24 May 10:59:56.103 # Failed opening .rdb for saving: Read-only file system
31602:M 24 May 10:59:56.204 # Background saving error
I'm using the defaults for the dir and dbfilename directives
# The filename where to dump the DB
dbfilename dump.rdb
# The working directory.
#
# The DB will be written inside this directory, with the filename specified
# above using the 'dbfilename' configuration directive.
#
# The Append Only File will also be created inside this directory.
#
# Note that you must specify a directory here, not a file name.
dir /var/lib/redis
user@server:/var/lib/redis$ ls -l
total 252
-rw-r--r-- 1 redis redis 249649 May 23 02:44 dump.rdb
It's owned by redis.. why would it be set to read-only?
ok... so
user@server:/var/lib/redis$ redis-cli config get dir
1) "dir"
2) "/var/spool/cron"
user@server:/var/lib/redis$ redis-cli config get dbfilename
1) "dbfilename"
2) "root"
This is starting to seem very strange. What on earth could have set it to that?