Searching through a large text or log file (10GB+)

user_6024530 0
Viewed 891

Part of a Python script that I'm writing requires me to find a particular string in a large text or log file: if it exists then do something; otherwise, do something else.

The files which are being fed in are extremely large (10GB+). It feels extremely slow and inefficient to use:

with open('file.txt') as f:
    for line in f:
        if some_string in line:
            return True
    return False

If the string doesn't exist in the file, then iterating through would take a long time.

Is there a time efficient way to achieve this?

file-read python
3 Answers

You can try with mmap:

>>> import mmap
>>> import re
>>> f = open("data.log", "r")
>>> mm = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
>>> re.search(b"test", mm)
<re.Match object; span=(12, 16), match=b'test'>
user_14166720

If you’re on Linux or BSD (Mac) I would just create a subprocess with grep or awk and let them do the search, they have had decades of optimisation for finding strings in big files. Make sure to include commandline flag to tell it to stop searching after the first match, if you only care that it exists and don’t need all instances or a count.

user_2967380

Try handling larger chunks instead of individual lines. For example:

def contains(filename, some_string):
    n = len(some_string)
    prev_chunk = ''
    with open(filename) as f:
        while chunk := f.read(2 ** 20):
            if some_string in prev_chunk[-(n-1):] + chunk:
                return True
            prev_chunk = chunk
        return False

I tried that with some made up 1 GB file and it took about 1 second to check a string that's not in there.

user_126710570
Related