Regex Expression Capture Existing and Non Existing Data

user_9582610 0
Viewed 48

I am trying to classiffy some logs to further analyze them in Python Pandas. A log sample would look like:

Event: Task_01Error:NO_ERROR
Event: Task_01Error:ERROR_MINOR
Event: Task_02Error:NO_ERROR
Event: Task_03Error:ERROR_01Details:BadData
Event: Task_03Error:ERROR_MINOR

I need to classify them as (my desired output):

Task, ErrorType,Details
01,NO_ERROR,NA
01,ERROR_MINOR,NA
02,NO_ERROR,NA
03,ERROR_01,BadData
03,ERROR_MINOR,NA

What I have come with so far is:

^Event: Task_(.*)Error:(.*)Details:(.*$)

Which only matches the fourth entry (as expected). But I actually need it to match either the information after the "Details:" string or nothing if the string is not there. The reason I need it that way is so I can use the regular expression with Pandas Series String Extract.

In other words, In the last group I need to match:

Details:(.*$)  OR ()

I know there must be an easy way to do this but I just cannot figure it out.

Thanks!

pandas regex-negation extract regex
3 Answers

Use assign function to create 2 new columns:

df.assign(
    ErrorType = lambda x: x['event'].apply(lambda s: s.split(':')[2].split('Details')[0]),
    Details = lambda x: x['event'].apply(lambda s: s.split('Details:')[1] if len(s.split('Details:'))>1 else 'NA')
)

You dont need regex as it is easy to extract data using split

user_39588780

You can also parse the log file upon reading.

import re
path_to_file = "/mnt/ramdisk/in.log"

# capture event and err-detail
patt = re.compile(r"""
    Event:\sTask_(\d+)
    Error:(.+)
    """, re.VERBOSE)

ls = []  # output container
with open(path_to_file) as f:
    for line in f:
        # 1. split task and err-detail
        match_obj = patt.match(line)
        ev = match_obj.group(1)
        # 2. split error and detail
        err_detail = match_obj.group(2).split("Details:")
        if len(err_detail) == 1:
            ls.append([ev, err_detail[0], None])  # or "NA"
        else:
            ls.append([ev, err_detail[0], err_detail[1]])

df = pd.DataFrame(ls, columns=["Task", "ErrorType", "Details"])

Result:

print(df)

  Task    ErrorType   Details
0   01     NO_ERROR      None
1   01  ERROR_MINOR      None
2   02     NO_ERROR      None
3   03     ERROR_01   BadData
4   03  ERROR_MINOR      None
user_32186930

For a pattern, you could use 3 capturing groups, where the third group is in an optional part.

If the 3rd group is not present, it can be N/A

^Event:\s+Task_(\d+)Error:(NO_ERROR|ERROR_(?:MINOR|\d+))(?:\w+:(\w+))?

Explanation

  • ^ Start of string
  • Event:\s+Task_ Match Event:\s+Task_
  • (\d+) Capture group 1, match 1+ digits
  • Error: Match literally
  • ( Capture group 2 =- NO_ERROR Match literally
    • | Or
    • ERROR_(?:MINOR|\d+) Match ERROR_MINOR or ERROR_ and 1+ digits
  • ) Close group 2
  • (?:\w+:(\w+))? Optionally match 1+ word chars and :, then capture in group 3 1+ word chars

Regex demo

import re
import pandas as pd

regex = r"^Event:\s+Task_(\d+)Error:(NO_ERROR|ERROR_(?:MINOR|\d+))(?:\w+:(\w+))?"

test_str = ("Event: Task_01Error:NO_ERROR\n"
            "Event: Task_01Error:ERROR_MINOR\n"
            "Event: Task_02Error:NO_ERROR\n"
            "Event: Task_03Error:ERROR_01Details:BadData\n"
            "Event: Task_03Error:ERROR_MINOR")

matches = re.findall(regex, test_str, re.MULTILINE)
df = pd.DataFrame(matches, columns=["Task", "ErrorType", "Details"])
print(df)

Output

 Task    ErrorType  Details
0   01     NO_ERROR         
1   01  ERROR_MINOR         
2   02     NO_ERROR         
3   03     ERROR_01  BadData
4   03  ERROR_MINOR
user_54249880
Related