Regex find groups and matches from a log file

user_12081172 0
Viewed 158

I have a log file with a lot of lines. Example:

219.133.7.154 - price5585 [21/Jun/2019:15:45:53 -0700] "GET /incubate/incubate HTTP/1.1" 201 12126

I need as the output something like this:

{host: 219.133.7.154, user: price5585, date: 21/Jun/2019:15:45:53 -0700, req: GET /incubate/incubate HTTP/1.1}

I really struggle with that and got only 2 first things working. Here is my code:

pattern = """
(?P<host>.*) #Host name
(-\ ) 
(?P<username>\w*)  #username

(?P<time>\w*) #Time

"""

How should pattern look, so I can extract everything I need?

python regex
3 Answers

Do you have to use regex? Because your goal can be easily achieved by parsing fields separated by whitespaces:

#!/usr/bin/env python3

LINE = "219.133.7.154 - price5585 [21/Jun/2019:15:45:53 -0700] \"GET /incubate/incubate HTTP/1.1\" 201 12126"

body = LINE.split("\"")[1]
split_line = LINE.split(" ")

output_dict = {"host": split_line[0],
               "user": split_line[2],
               "date": " ".join([split_line[3], split_line[4]]),
               "req": body}
print(output_dict)

Output:

{'host': '219.133.7.154', 'user': 'price5585', 'date': '[21/Jun/2019:15:45:53 -0700]', 'req': 'GET /incubate/incubate HTTP/1.1'}
user_36918910

With the use of regex:

import re
line = '219.133.7.154 - price5585 [21/Jun/2019:15:45:53 -0700] "GET /incubate/incubate HTTP/1.1" 201 12126'
pat = r'(\S+) - (\S+) \[(.+)\] "(.+)"'
m = re.match(pat, line)
dic = {'host': m.group(1), 'user': m.group(2), \
    'date': m.group(3), 'req': m.group(4)}
print(dic)

{'host': '219.133.7.154', 'user': 'price5585', 'date': '21/Jun/2019:15:45:53 -0700', 'req': 'GET /incubate/incubate HTTP/1.1'}
user_64014030

You can omit this part as a capture group (-\ ) and while this part (?P<host>.*) works for the given example data, the .* can be non greedy preventing some backtracking as it will first match until the end of the string.

This group (?P<time>\w*) does not match anything, as the next text to match is [21/Jun/2019:15:45:53 -0700] and \w* matches optional word characters.

You could write the pattern as:

(?P<host>.*?) - (?P<username>\w+) \[(?P<date>[^][]*)] "(?P<req>[^"]+)"

Regex demo | Python demo

For example

import re

pattern = r"(?P<host>.*?) - (?P<username>\w+) \[(?P<date>[^][]*)] \"(?P<req>[^\"]+)\""
s = "219.133.7.154 - price5585 [21/Jun/2019:15:45:53 -0700] \"GET /incubate/incubate HTTP/1.1\" 201 12126"
m = re.match(pattern, s)

if m:
    print(m.groupdict())

Output

{'host': '219.133.7.154', 'username': 'price5585', 'date': '21/Jun/2019:15:45:53 -0700', 'req': 'GET /incubate/incubate HTTP/1.1'}
user_54249880
Related