I am using docker-compose with four postgres containers a redis_master container and a redis_slave container. The redis server boots normally but about every 8 hours after launch the slave cannot reach the master for replication. The docker-compose logs show the following error logs from redis_master repeating:
redis_master | 1:S 15 Jul 2020 11:39:05.338 * Connecting to MASTER UNKNOWN.IP:58270
redis_master | 1:S 15 Jul 2020 11:39:05.338 * MASTER REPLICA sync started
redis_master | 1:S 15 Jul 2020 11:39:05.497 # Error condition on socket for SYNC: Connection refused
redis_master | 1:S 15 Jul 2020 11:39:06.341 * Connecting to MASTER UNKNOWN.IP:58270
redis_master | 1:S 15 Jul 2020 11:39:06.341 * MASTER REPLICA sync started
redis_master | 1:S 15 Jul 2020 11:39:06.506 # Error condition on socket for SYNC: Connection refused
The UNKNOWN.IP is not a private IP address, not my server IP nor the IP of any clients accessing the server. That IP address is first used in the logs here:
redis_master | 1:S 15 Jul 2020 11:37:50.127 * REPLICAOF UNKNOWN.IP:58270 enabled (user request from 'id=525 addr=UNKNOWN.IP:35762 fd=13 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=48 qbuf-free=32720 obl=0 oll=0 omem=0 events=r cmd=slaveof')
This seems to trigger an error when trying to interact with the redis server.
WARN [2020-07-15 13:58:42,036] org.eclipse.jetty.server.HttpChannel: /v1/websocket/
! redis.clients.jedis.exceptions.JedisDataException: READONLY You can't write against a read only replica.
The server has also been hit by a crypto mining malware "kdevtmpfsi" which seems to have gotten into the redis server, possibly causing some of these issues. The malware is using about 400% of CPU and 1GB of memory. I have not been able to get rid of it completely and I am trying the process on a new server with more ports closed to try and stop the malware from getting in again. Any advice for stopping the malware from getting in again? Or what might be causing the redis replication issue. My docker-compose config file is taken from this GitHub repo for cloning the Signal app's server. I start the docker environment with
sudo docker-compose up
and I am using all default configurations for redis.
docker-compose.yml
version: '2.2'
services:
signal_account_database:
image: postgres:11
container_name: postgres_account_database
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: signal
PGDATA: /var/lib/postgresql/data/pgdata
ports:
- '5431:5432'
volumes:
- ./postgres_database:/var/lib/postgresql/data
signal_keys_database:
image: postgres:11
container_name: postgres_keys_database
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: signal
PGDATA: /var/lib/postgresql/data/pgdata
ports:
- '5432:5432'
volumes:
- ./postgres_keys_database:/var/lib/postgresql/data
signal_message_database:
image: postgres:11
container_name: postgres_message_database
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: signal
PGDATA: /var/lib/postgresql/data/pgdata
ports:
- '5433:5432'
volumes:
- ./postgres_message_store:/var/lib/postgresql/data
signal_abuse_database:
image: postgres:11
container_name: postgres_abuse_database
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: signal
PGDATA: /var/lib/postgresql/data/pgdata
ports:
- '5434:5432'
volumes:
- ./postgres_abuse_database:/var/lib/postgresql/data
redis_main:
image: redis:5
container_name: redis_master
ports:
- '6379:6379'
volumes:
- ./redis_main:/data
redis_replication:
image: redis:5
container_name: redis_slave
command: redis-server --port 6380
ports:
- '6380:6380'
volumes:
- ./redis_replication:/data
Has anyone else had the same replication problem? Looking for advice on fixing this issue.