redis_slave can't reach redis_master when using docker-compose

Viewed 1116

I am using docker-compose with four postgres containers a redis_master container and a redis_slave container. The redis server boots normally but about every 8 hours after launch the slave cannot reach the master for replication. The docker-compose logs show the following error logs from redis_master repeating:

redis_master               | 1:S 15 Jul 2020 11:39:05.338 * Connecting to MASTER UNKNOWN.IP:58270
redis_master               | 1:S 15 Jul 2020 11:39:05.338 * MASTER  REPLICA sync started
redis_master               | 1:S 15 Jul 2020 11:39:05.497 # Error condition on socket for SYNC: Connection refused
redis_master               | 1:S 15 Jul 2020 11:39:06.341 * Connecting to MASTER UNKNOWN.IP:58270
redis_master               | 1:S 15 Jul 2020 11:39:06.341 * MASTER  REPLICA sync started
redis_master               | 1:S 15 Jul 2020 11:39:06.506 # Error condition on socket for SYNC: Connection refused

The UNKNOWN.IP is not a private IP address, not my server IP nor the IP of any clients accessing the server. That IP address is first used in the logs here:

redis_master               | 1:S 15 Jul 2020 11:37:50.127 * REPLICAOF UNKNOWN.IP:58270 enabled (user request from 'id=525 addr=UNKNOWN.IP:35762 fd=13 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=48 qbuf-free=32720 obl=0 oll=0 omem=0 events=r cmd=slaveof')

This seems to trigger an error when trying to interact with the redis server.

WARN  [2020-07-15 13:58:42,036] org.eclipse.jetty.server.HttpChannel: /v1/websocket/
! redis.clients.jedis.exceptions.JedisDataException: READONLY You can't write against a read only replica.

The server has also been hit by a crypto mining malware "kdevtmpfsi" which seems to have gotten into the redis server, possibly causing some of these issues. The malware is using about 400% of CPU and 1GB of memory. I have not been able to get rid of it completely and I am trying the process on a new server with more ports closed to try and stop the malware from getting in again. Any advice for stopping the malware from getting in again? Or what might be causing the redis replication issue. My docker-compose config file is taken from this GitHub repo for cloning the Signal app's server. I start the docker environment with

sudo docker-compose up

and I am using all default configurations for redis.

docker-compose.yml


version: '2.2'
services:
  signal_account_database:
    image:          postgres:11
    container_name: postgres_account_database
    restart:        always
    environment:
      POSTGRES_USER:     postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB:       signal
      PGDATA:            /var/lib/postgresql/data/pgdata
    ports:
      - '5431:5432'
    volumes:
      - ./postgres_database:/var/lib/postgresql/data
  signal_keys_database:
    image:          postgres:11
    container_name: postgres_keys_database
    restart:        always
    environment:
      POSTGRES_USER:     postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB:       signal
      PGDATA:            /var/lib/postgresql/data/pgdata
    ports:
      - '5432:5432'
    volumes:
      - ./postgres_keys_database:/var/lib/postgresql/data
  signal_message_database:
    image:          postgres:11
    container_name: postgres_message_database
    restart:        always
    environment:
      POSTGRES_USER:     postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB:       signal
      PGDATA:            /var/lib/postgresql/data/pgdata
    ports:
      - '5433:5432'
    volumes:
      - ./postgres_message_store:/var/lib/postgresql/data
  signal_abuse_database:
    image:          postgres:11
    container_name: postgres_abuse_database
    restart:        always
    environment:
      POSTGRES_USER:     postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB:       signal
      PGDATA:            /var/lib/postgresql/data/pgdata
    ports:
      - '5434:5432'
    volumes:
      - ./postgres_abuse_database:/var/lib/postgresql/data
  redis_main:
    image: redis:5
    container_name: redis_master
    ports:
      - '6379:6379'
    volumes:
      - ./redis_main:/data
  redis_replication:
    image: redis:5
    container_name: redis_slave
    command: redis-server --port 6380
    ports:
      - '6380:6380'
    volumes:
      - ./redis_replication:/data

Has anyone else had the same replication problem? Looking for advice on fixing this issue.

1 Answers
redis_main:
    image: redis:5
    container_name: redis_master
    restart: always
    command: redis-server --port 6379
    ports:
      - '6379:6379'
    volumes:
      - ./redis_main:/data
  redis_replication:
    image: redis:5
    container_name: redis_slave
    command: redis-server --slaveof 127.0.0.1 6379 --port 6380 --slave-announce-ip 127.0.0.1
    ports:
      - '6380:6380'
    volumes:
      - ./redis_replication:/data


  sentinel:
    build: ./sentinel
    container_name: redis_sentinel
    restart: always
    command: redis-sentinel --port 26379
    ports:
      - '26379:26379'
    environment:
      - SENTINEL_NAME=mysentinel
      - HOST_IP=127.0.0.1
    volumes:
      - ./redis_slave:/data

Related