Python Logs to Logstash(ELK) issue

Question

having issue with sending logs from Python to Logstash(ELK stack).

I have a docker configuration for ELK like this:

version: '3.2'

services:
...

  logstash:
    restart: always
    container_name: logstash
    build:
      context: logstash/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./logstash/config/logstash.yml
        target: /usr/share/logstash/config/logstash.yml
        read_only: true
      - type: bind
        source: ./logstash/pipeline
        target: /usr/share/logstash/pipeline
        read_only: true
    ports:
      - "5010:5000"
      - "5001:5001"
      - "5002:5002"
      - "5003:5003"
      - "9600:9600"
      - "5959:5959"
      - "12201:12201/udp"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - backendnetwork
    depends_on:
      - elasticsearch

  kibana:
...

  apm-server:
....

networks:
  backendnetwork:
    driver: bridge

And a Logstash config:

input {
        tcp {
                port => 5000
        }
        gelf {
                port => 12201
                type => gelf
        }
        tcp {
                port => 5001
                type => "python"
                codec => "json_lines"
        }
        tcp {
                port => 5002
                type => "python2"
                codec => "json_lines"
        }
        tcp {
                port => 5003
                type => "python3"
                codec => "json_lines"
        }
    tcp {
            port => 5959
            type => "python_logs"
            codec => json
    }

}

## Add your filters / logstash plugins configuration here

output {
    if [type] == "nginx" {
        ...
    }
    else if [type] == "gelf" {
        ...
    }

    else if [type] == "python" {
       ...
    }
    else if [type] == "python_logs" {
        elasticsearch {
                hosts => "elasticsearch:9200"
                user => "elastic"
        password => "password"
                index    => "python_logs-%{+YYYY.MM.dd}"
        }
    }
    else if [type] == "python2" {
        ...
    }
    else if [type] == "python3" {
        ...
    }

    else {
        elasticsearch {
                hosts => "elasticsearch:9200"
                user => "elastic"
        password => "wnl8bFxsJZpgRsCfyejJPGJK"
                index    => "other-%{+YYYY.MM.dd}"
        }
    }
}

I expose port 5959 for json format to receive logs from Python. It is configured in main docker-compose to propagate 5959 port from Docker container.

When I send HTTP request to logstash using Postman:

Postman screenshot

Then I can see in logstash logs that it was receiving a message:

 at [Source: (String)"PUT /tweeter/tweet HTTP/1.1
User-Agent: PostmanRuntime/7.29.0
Accept: */*
Postman-Token: 9520b257-94e4-437d-a57e-02bfc9394180
Host: 127.0.0.1:5959
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 13

message=Hello"; line: 1, column: 4]>, :data=>"PUT /tweeter/tweet HTTP/1.1
User-Agent: PostmanRuntime/7.29.0
Accept: */*
Postman-Token: 9520b257-94e4-437d-a57e-02bfc9394180
Host: 127.0.0.1:5959
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 13

message=Hello"}

Jumping to Python. I have this test code:

import logging

import logstash

host = 'localhost'
test_logger = logging.getLogger('python-logstash-logger')
test_logger.setLevel(logging.INFO)
test_logger.addHandler(logstash.TCPLogstashHandler(host, 5959, version=1))

def run_app():
    test_logger.info('INFO: Test Log')
    test_logger.debug('DEBUG: Test Log')
    test_logger.error('ERROR: Test Log')


if __name__ == '__main__':
    run_app()

When I run my test code, I can see that logger was configured and handler is assigned to the logger: Intellij Idea Debug screenshot

However, after I run my app, I can not see any logs were pushed to logstash. There is only 1 entry related to that I pushed through Postman: Kibana view of index logs

Could you guys help me to understand why logs from Python do not go to the Logstash?

Python Logs to Logstash(ELK) issue

0 Answers