I have conf file in logstash like this
input {
beats {
host => "0.0.0.0"
port => 5044
ssl => false
}
}
filter {
if [fileset][name] == "auth" {
grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?"]}
add_field => [ "activity", "SSH Login" ]
add_tag => ["linux_auth"]
}
date {
match => [ "[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
geoip {
source => "[system][auth][ssh][ip]"
target => "[system][auth][ssh][geoip]"
}
}
else if [fileset][name] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_tag => ["linux_syslog"]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => ["192.168.186.157:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
I send auth.log and syslog to logstash. I need log like
Jul 16 06:59:32 kali sshd[5835]: Failed password for kali from 192.168.186.149 port 54780 ssh2
to search brute force attack. But when I see Kibana every line like that disappeared. And syslog didn't appear too.
Here is some auth.log for you to check
Jul 16 06:58:57 kali sshd[5810]: PAM 3 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:58:57 kali sshd[5810]: PAM service(sshd) ignoring max retries; 4 > 3
Jul 16 06:58:57 kali sshd[5828]: Failed password for kali from 192.168.186.149 port 54778 ssh2
Jul 16 06:58:57 kali sshd[5828]: Connection closed by authenticating user kali 192.168.186.149 port 54778 [preauth]
Jul 16 06:58:57 kali sshd[5828]: PAM 3 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:58:57 kali sshd[5828]: PAM service(sshd) ignoring max retries; 4 > 3
Jul 16 06:58:59 kali sshd[5802]: Failed password for kali from 192.168.186.149 port 54752 ssh2
Jul 16 06:58:59 kali sshd[5801]: Failed password for kali from 192.168.186.149 port 54750 ssh2
Jul 16 06:58:59 kali sshd[5799]: Failed password for kali from 192.168.186.149 port 54746 ssh2
Jul 16 06:58:59 kali sshd[5803]: Failed password for kali from 192.168.186.149 port 54754 ssh2
Jul 16 06:58:59 kali sshd[5800]: Failed password for kali from 192.168.186.149 port 54748 ssh2
Jul 16 06:58:59 kali sshd[5804]: Failed password for kali from 192.168.186.149 port 54756 ssh2
Jul 16 06:58:59 kali sshd[5798]: Failed password for kali from 192.168.186.149 port 54744 ssh2
Jul 16 06:58:59 kali sshd[5805]: Failed password for kali from 192.168.186.149 port 54758 ssh2
Jul 16 06:58:59 kali sshd[5806]: Failed password for kali from 192.168.186.149 port 54760 ssh2
Jul 16 06:58:59 kali sshd[5812]: Failed password for kali from 192.168.186.149 port 54774 ssh2
Jul 16 06:58:59 kali sshd[5807]: Failed password for kali from 192.168.186.149 port 54762 ssh2
Jul 16 06:58:59 kali sshd[5809]: Failed password for kali from 192.168.186.149 port 54766 ssh2
Jul 16 06:58:59 kali sshd[5808]: Failed password for kali from 192.168.186.149 port 54764 ssh2
Jul 16 06:58:59 kali sshd[5811]: Failed password for kali from 192.168.186.149 port 54772 ssh2
Jul 16 06:59:00 kali sshd[5802]: Connection closed by authenticating user kali 192.168.186.149 port 54752 [preauth]
Jul 16 06:59:00 kali sshd[5802]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5802]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:00 kali sshd[5801]: Connection closed by authenticating user kali 192.168.186.149 port 54750 [preauth]
Jul 16 06:59:00 kali sshd[5801]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5801]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:00 kali sshd[5799]: Connection closed by authenticating user kali 192.168.186.149 port 54746 [preauth]
Jul 16 06:59:00 kali sshd[5799]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5799]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:00 kali sshd[5800]: error: maximum authentication attempts exceeded for kali from 192.168.186.149 port 54748 ssh2 [preauth]
Jul 16 06:59:00 kali sshd[5800]: Disconnecting authenticating user kali 192.168.186.149 port 54748: Too many authentication failures [preauth]
Jul 16 06:59:00 kali sshd[5800]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5800]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:00 kali sshd[5803]: Connection closed by authenticating user kali 192.168.186.149 port 54754 [preauth]
Jul 16 06:59:00 kali sshd[5803]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5803]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:00 kali sshd[5804]: Connection closed by authenticating user kali 192.168.186.149 port 54756 [preauth]
Jul 16 06:59:00 kali sshd[5804]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5804]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:00 kali sshd[5798]: Connection closed by authenticating user kali 192.168.186.149 port 54744 [preauth]
Jul 16 06:59:00 kali sshd[5798]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5798]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:00 kali sshd[5805]: Connection closed by authenticating user kali 192.168.186.149 port 54758 [preauth]
Jul 16 06:59:00 kali sshd[5805]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5805]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:00 kali sshd[5806]: Connection closed by authenticating user kali 192.168.186.149 port 54760 [preauth]
Jul 16 06:59:00 kali sshd[5806]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5806]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:00 kali sshd[5812]: Connection closed by authenticating user kali 192.168.186.149 port 54774 [preauth]
Jul 16 06:59:00 kali sshd[5812]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5812]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:00 kali sshd[5807]: Connection closed by authenticating user kali 192.168.186.149 port 54762 [preauth]
Jul 16 06:59:00 kali sshd[5807]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5807]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:00 kali sshd[5809]: Connection closed by authenticating user kali 192.168.186.149 port 54766 [preauth]
Jul 16 06:59:00 kali sshd[5809]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5809]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:00 kali sshd[5811]: Connection closed by authenticating user kali 192.168.186.149 port 54772 [preauth]
Jul 16 06:59:00 kali sshd[5811]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5811]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:00 kali sshd[5808]: Connection closed by authenticating user kali 192.168.186.149 port 54764 [preauth]
Jul 16 06:59:00 kali sshd[5808]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:00 kali sshd[5808]: PAM service(sshd) ignoring max retries; 5 > 3
Jul 16 06:59:29 kali sshd[2186]: exited MaxStartups throttling after 00:00:44, 2 connections dropped
Jul 16 06:59:29 kali sshd[5835]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.186.149 user=kali
Jul 16 06:59:32 kali sshd[5835]: Failed password for kali from 192.168.186.149 port 54780 ssh2
Jul 16 06:59:32 kali sshd[5835]: Connection closed by authenticating user kali 192.168.186.149 port 54780 [preauth]
Jul 16 07:05:01 kali CRON[5840]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 16 07:05:01 kali CRON[5840]: pam_unix(cron:session): session closed for user root
And this is the message in Kibana, auth.log lost some data with tag _grokparsefailure, _geoip_lookup_failure, and syslog lost all of it.
