Docker containers not logging to loki using loki log driver

Question

I'm trying to use the loki log driver in a docker container on an AWS ec2 host. The config below works fine on my own ubuntu 20.04 machine but not in an ubuntu 16.04 ec2 host. iptables has a loopback rule and the appropriate docker port for loki, :3100. I even opened up that port in the security group. The docker version is 20.10.2.

I have tried to use localhost:3100, the hostname:3100, also the docker container name. I have a bridge network and don't want to use a host network. I also don't want to use the container IP address.

Nothing in this SO question works for me. How to access host port from docker container

Here is my docker-compose.yaml which works on my local machine but not in ec2.

the tools container is a go html server. The tools and loki containers are on the same host.

I can post a log to loki from an alpine container using curl to URL http://loki:3100/loki/api/v1/push but not from my tools container. I can connect with telnet but curl times out.

version: '3.3'

networks:
  traefik:
    external: true
  loki: {}
  
services:

  tools:
    build: .
    container_name: tools
    restart: always
    networks:
      - traefik
      - loki
    extra_hosts:
      - "host.docker.internal:host-gateway"
    logging:
      driver: loki
      options: 
        loki-url: http://loki:3100/loki/api/v1/push
        loki-external-labels: job=tools
    labels:
    ...

Here is my loki docker-compose.yaml

version: "3.8"

networks:
  traefik:
    external: true
  loki: {}

volumes:
  loki_data:
   
services:
  
  loki:
    container_name: "loki"
    image: grafana/loki:2.1.0
    restart: always
    networks:
      - traefik
      - loki
    ports:
      - 3100:3100
    volumes:
      - type: volume
        source: loki_data
        target: /data
      - type: bind
        source: ./config/s3-loki-bolt-conf.yml
        target: /etc/loki/local-config.yaml
    command: -config.file=/etc/loki/local-config.yaml        
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

the docker plugin is installed

docker plugin ls
ID             NAME          DESCRIPTION           ENABLED
a03c22e8375e   loki:latest   Loki Logging Driver   true

sudo journalctl -u docker.service | grep loki is giving me this error.

08 06:46:06 docker1 dockerd[30842]: time="2021-02-08T06:46:06-08:00" level=info msg="level=warn ts=2021-02-08T14:46:06.586642758Z caller=client.go:322 container_id=5a5fbd8a7077243de9db74f549ab619f783eda978ee234651ad1849263a534fe component=client host=localhost:3100 msg=\"error sending batch, will retry\" status=-1 error=\"Post \\"http://localhost:3100/loki/api/v1/push\\": context deadline exceeded\"" plugin=fef8734ec8cc2d252f8c4e73e3e91fe8293d2847c7ce1d6df2fb2172a1c288ce

Docker containers not logging to loki using loki log driver

0 Answers