Django server not sending logs to Logstash

user_5250746 0
Viewed 5270

I am using ELK stack for centralised logging from my Django server. My ELK stack is on a remote server and logstash.conf looks like this:

input {
    tcp {
    port => 5959
    codec => json
  }
}
output {
  elasticsearch {
    hosts => ["xx.xx.xx.xx:9200"]
  }
}

Both services elasticsearch and logstash are working (checked using docker-compose logs logstash).

My Django server's settings file has logging configured as below:

LOGGING = {
  'version': 1,
  'handlers': {
        'logstash': {
            'level': 'INFO',
            'class': 'logstash.TCPLogstashHandler',
            'host': 'xx.xx.xx.xx',
            'port': 5959, # Default value: 5959
            'version': 0, # Version of logstash event schema. Default value: 0 (for backward compatibility of the library)
            'message_type': 'django',  # 'type' field in logstash message. Default value: 'logstash'.
            'fqdn': True, # Fully qualified domain name. Default value: false.
            'tags': ['django.request'], # list of tags. Default: None.
        },
  },
  'loggers': {
        'django.request': {
            'handlers': ['logstash'],
            'level': 'DEBUG',
  },
}
}

I run my Django server and Logstash handler handles the logs as console shows no logs. I used the python-logstash library in Django server to construct the above conf, but the logs are not sent to my remote server.

I checked through many questions, verified that services are running and ports are correct, but I have no clue why the logs are not being sent to Logstash.

elastic-stack logstash django python
4 Answers

I faced the same issue but after scrambling around a lot, I made it work by changing django.request to django.server. I got the idea when I used only django as a logger name in the python code and then found out the actual logger_name from log data stored in elasticsearch. Below is the updated code

LOGGING = {
  'version': 1,
  'handlers': {
        'logstash': {
            'level': 'INFO',
            'class': 'logstash.TCPLogstashHandler',
            'host': 'xx.xx.xx.xx',
            'port': 5959, # Default value: 5959
            'version': 0, # Version of logstash event schema. Default value: 0 (for backward compatibility of the library)
            'message_type': 'django',  # 'type' field in logstash message. Default value: 'logstash'.
            'fqdn': True, # Fully qualified domain name. Default value: false.
            'tags': ['django.request'], # list of tags. Default: None.
        },
  },
  'loggers': {
        'django.server': {  # Here is the change
            'handlers': ['logstash'],
            'level': 'DEBUG',
      }
  },
}

Refer to this for more detail on django logging https://docs.djangoproject.com/en/1.11/topics/logging/#id3

user_53572870

Your logging configuration is correct. You need to mention index name in elasticsearch config part of logstash conf. Update your logstash config to 

input {
    tcp {
    port => 5959
    codec => json
  }
}
output {
  elasticsearch {
    hosts => ["xx.xx.xx.xx:9200"]
    manage_template => false
    index => "djangologs"
  }
}

If you are using google cloud or AWS, open port/update firewall rules.

user_15226610

Thank you Tasneem,

This worked on my machine:

mysite/settings.py

LOGGING = {
  'version': 1,
  'handlers': {
      'logstash': {
          'level': 'DEBUG',
          'class': 'logstash.TCPLogstashHandler',
          'host': 'xx.xx.xx.xx',
          'port': 9600, # Default value: 5959
          'version': 1, # Version of logstash event schema. Default value: 0 (for backward compatibility of the library)
          'message_type': 'django',  # 'type' field in logstash message. Default value: 'logstash'.
          'fqdn': False, # Fully qualified domain name. Default value: false.
          'tags': ['django.request'],# list of tags. Default: None.
      },
  },
  'loggers': {
      'django.server': {
          'handlers': ['logstash'],
          'level': 'DEBUG',
      },
  },
}

logstash# cat logstash-django.conf

input {
  tcp {
    port => 9600
    codec => json
  }
}
output {
    elasticsearch {
      hosts => ["https://your.elasticsearch.com:port"]
      user => ["xx"]
      password => ["xx"]
      index => "djangotest"
      document_type => "%{type}"
    }
   stdout {
      codec => rubydebug {metadata => true }
  }
}
user_58150860
Related