Grok pattern matching custom junos log

ChrisChoBG 0
Viewed 31

I'm pretty new to grok and regex at all. I want to match custom junos log. Log examples:

  RT_FLOW_SESSION_CREATE: session created 192.168.1.1/22->192.168.200.220/4 0x0 icmp 100.101.102.103/7090->190.190.190.220/4 0x0 source rule nat_to_external N/A N/A 1
  RT_FLOW_SESSION_CREATE: session created 1.1.1.11/6500->1.168.2.3/143 0x0 junos-https 11.67.11.11/15020->12.18.8.7/43 0x0 N/A N/A N/A N/A 6 
  RT_FLOW_SESSION_CREATE: session created 19.41.43.10/5280->92.68.8.19/1000 0x0 None 98.41.43.10/5280->12.68.8.19/200 0x0 N/A N/A destination rule ServerDEV 6

So as you can see in above examples N/A is changing, i hope you can suggest how to match N/A in all cases(N/A = N/A, source rule, nat_to_external, destination rule, ServerDEV) with grok or regex.

I've created this pattern(please notice that this is full pattern and above example , but it's not consistent with all cases:

:%{GREEDYDATA:session}: %{DATA:reason} %{IP:src_ip}/%{NUMBER:src_port}->%{IP:dst_ip}/%{NUMBER:dst_port} 0x%{BASE10NUM:0xconnection_tag} %{JUNOS_SERVICE:service_name} %{IP:nat_src_ip}/%{NUMBER:nat_src_port}->%{IP:nat_dst_ip}/%{NUMBER:nat_dst_port} 0x%{BASE10NUM:0xnat-connection_tag} %{JUNOS_NAT_RULE:src_nat_rule_type} %{JUNOS_NAT_RULE:src_nat_rule_name} %{JUNOS_NAT_RULE:dst_nat_rule_type} %{JUNOS_NAT_RULE:dst_nat_rule_name} %{NUMBER:protocol_id} %{WORD:policy_name} %{DATA:src_zone_name} %{DATA:dst_zone_name} %{NUMBER:session-id} N/A\(N/A\) %{JUNOS_INTERFACE:packet_incoming_interface} %{WORD:application} %{WORD:nested_application} %{DATA:encrypted} %{DATA:application-category} %{DATA:application-sub-category} %{NUMBER:application_risk} %{DATA:application-characteristics} N/A N/A

Referenced Patterns:

JUNOS_REASON ([\s\w\/-]+)
JUNOS_SERVICE (\b[\w\/-]+\b)
WORD (\b[\w\/-]+\b)
JUNOS_NAT_RULE ([\w\/-]+)
JUNOS_INTERFACE (\b[\w0-9\.\/-]+\b)

I need to extract those logs in Graylog and my workaround for now is to use regex for cases where is not N/A.

i.e N/A is source rule

source rule (([^\s]+))

But i will be glad if you know more elegant solution.

grok regex
1 Answers

There is a slight change in 1st sentence log rather than others. Hence we have to use two grok patterns.

For 1st sentence the grok should be:

%{DATA:session_flow}: %{DATA:session_action} %{IP:session_ip}/%{NUMBER:session_port}->%{IP:forwarded_session}/%{NUMBER:forwarded_port} %{DATA:session_bytes} %{DATA:protocol} %{IP:destination_ip}/%{NUMBER:port}->%{IP:forwarded_destination_ip}/%{NUMBER:forwarded_destination_port} %{DATA:dest_session_bytes} (?<source_rule>%{WORD} %{WORD}) %{DATA:nat_to_external} %{DATA:destination_rule} %{DATA:ServerDEV} %{NUMBER:bytes}

For other sentences the grok is:

%{DATA:session_flow}: %{DATA:session_action} %{IP:session_ip}/%{NUMBER:session_port}->%{IP:forwarded_session}/%{NUMBER:forwarded_port} %{DATA:session_bytes} %{DATA:protocol} %{IP:destination_ip}/%{NUMBER:port}->%{IP:forwarded_destination_ip}/%{NUMBER:forwarded_destination_port} %{DATA:dest_session_bytes} %{DATA:source_rule} %{DATA:nat_to_external} %{DATA:destination_rule} %{DATA:ServerDEV} %{NUMBER:bytes}

Hence, Combining these two grok pattern ,You can try the below in logstash

grok
{

match => 
{
"message" => ['%{DATA:session_flow}: %{DATA:session_action} %{IP:session_ip}/%{NUMBER:session_port}->%{IP:forwarded_session}/%{NUMBER:forwarded_port} %{DATA:session_bytes} %{DATA:protocol} %{IP:destination_ip}/%{NUMBER:port}->%{IP:forwarded_destination_ip}/%{NUMBER:forwarded_destination_port} %{DATA:dest_session_bytes} (?<source_rule>%{WORD} %{WORD}) %{DATA:nat_to_external} %{DATA:destination_rule} %{DATA:ServerDEV} %{NUMBER:bytes}', '%{DATA:session_flow}: %{DATA:session_action} %{IP:session_ip}/%{NUMBER:session_port}->%{IP:forwarded_session}/%{NUMBER:forwarded_port} %{DATA:session_bytes} %{DATA:protocol} %{IP:destination_ip}/%{NUMBER:port}->%{IP:forwarded_destination_ip}/%{NUMBER:forwarded_destination_port} %{DATA:dest_session_bytes} %{DATA:source_rule} %{DATA:nat_to_external} %{DATA:destination_rule} %{DATA:ServerDEV} %{NUMBER:bytes}']
}

}

Keep Posted on how it goes!!! Thanks !!!

sudhagar ramesh0
Related