How to collect all lines of data between keywords in a file - starting+ending at linebreaks

user_10626011 0
Viewed 150

I am trying to collect specific information from very large log files but cannot figure out how to get the behavior I need.

For reference, an example log is sort of like this:

garbage I don't need - garbage I don't need
timestamp - date - server info - 'keyword 1' - data
more data more data more data more data
more data more data more data more data
more data more data 'keyword 2' - last bit of data
garbage I don't need - garbage I don't need

What I need is to find 'keyword 1', grab the whole line keyword 1 is on (back to timestamp) and all subsequent lines until (and including) the whole line that 'keyword 2' is on (through the last bit of data).

So far I have tried a few things. I cannot get decent results with re methods (findall, match, search etc.); I cannot figure out how to grab the data before the match (even with a look behind) but more importantly, I cannot figure out how to have the capture stop at a phrase and not just a single character.

for match in re.findall('keyword1[keyword2]+|', showall.read()):

I also tried something like this:

start_capture = False
for current_line in fileName:
    if 'keyword1' in current_line:
        start_capture = True
    if start_capture:
        new_list.append(current_line)
    if 'keyword2' in current_line:
        return(new_list)

No matter what I tried, this returned an empty list

Finally,I tried something like this:

def takewhile_plus_next(predicate, xs):
for x in xs:
    if not predicate(x):
        break
    yield x
yield x
with lastdb as f:
    lines = map(str.rstrip, f)
    skipped = dropwhile(lambda line: 'Warning: fatal assert' not in line, lines)
    lines_to_keep = takewhile_plus_next(lambda line: 'uptime:' not in line, skipped)

This last one took everything from keyword 1 to the EOF which includes almost 100,000 lines of garbage data.

python-3.x parsing python regex
3 Answers

You can use regex if you specify re.dotall and use lazy anythings .*? to match start and end:

import re

regex = r"\n.*?(keyword 1).*?(keyword 2).*?$"

test_str = ("garbage I don't need - garbage I don't need\n"
    "timestamp - date - server info - 'keyword 1' - data\n"
    "more data more data more data more data\n"
    "more data more data more data more data\n"
    "more data more data 'keyword 2' - last bit of data\n"
    "garbage I don't need - garbage I don't need")

matches = re.finditer(regex, test_str, re.DOTALL | re.MULTILINE)

for matchNum, match in enumerate(matches):
    matchNum = matchNum + 1

    print (match.group()) # your match is the whole group

Output:

timestamp - date - server info - 'keyword 1' - data 
more data more data more data more data
more data more data more data more data
more data more data 'keyword 2' - last bit of data

You might need to strip('\n') from it ...

You can view it here: https://regex101.com/r/HWIALZ/1 - it also holds the explanation of the patter. The short of it:

\n        newline 
   .*?    as few as possible anythings
   (keyword 1)   literal text - the () are not needed only if you want the group
   .*?    as few as possible anythings
   (keyword 2)   literal text - again () are not needed 
   .*?    as few as possible anythings
$         end of line

I included the () for clarity - you do not evaluate groups, you you remove them.

user_75053950

The following is fast for any size of file. It extracts from a 250M log file of nearly 2 million lines in 3 seconds. The extracted portion was at the end of the file.

I would not recommend using list, regexes or other in-memory techniques if there is a chance your files won't fit in available memory.

Test text file startstop_text:

line 1 this should not appear in output
line 2 keyword1
line 3 appears in output
line 4 keyword2
line 5 this should not appear in output

Code:

from itertools import dropwhile


def keepuntil(contains_end_keyword, lines):
    for line in lines:
        yield line
        if contains_end_keyword(line):
            break


with open('startstop_text', 'r') as f:
    from_start_line = dropwhile(lambda line: 'keyword1' not in line, f)
    extracted = keepuntil(lambda line: 'keyword2' in line, from_start_line)
    for line in extracted:
        print(line.rstrip())


>>> python startstop.py
line 2 keyword1
line 3 appears in output
line 4 keyword2
user_24809470

None of the other responses worked but I was able to figure it out using regex. 

for match in re.findall(".*keyword1[\s\S]*?keyword2:[\s\S]*?keyword3.*", log_file.read()):
user_106260110
Related