Cheap way to search a large text file for a string

The following function works for textfiles and binary files (returns only position in byte-count though), it does have the benefit to find strings even if they would overlap a line or buffer and would not be found when searching line- or buffer-wise.

def fnd(fname, s, start=0):
    with open(fname, 'rb') as f:
        fsize = os.path.getsize(fname)
        bsize = 4096
        buffer = None
        if start > 0:
            f.seek(start)
        overlap = len(s) - 1
        while True:
            if (f.tell() >= overlap and f.tell() < fsize):
                f.seek(f.tell() - overlap)
            buffer = f.read(bsize)
            if buffer:
                pos = buffer.find(s)
                if pos >= 0:
                    return f.tell() - (len(buffer) - pos)
            else:
                return -1

The idea behind this is:

• seek to a start position in file
• read from file to buffer (the search strings has to be smaller than the buffer size) but if not at the beginning, drop back the - 1 bytes, to catch the string if started at the end of the last read buffer and continued on the next one.
• return position or -1 if not found

I used something like this to find signatures of files inside larger ISO9660 files, which was quite fast and did not use much memory, you can also use a larger buffer to speed things up.

I like the solution of Javier. I did not try it, but it sounds cool!

For reading through a arbitary large text and wanting to know it a string exists, replace it, you can use Flashtext, which is faster than Regex with very large files.

Edit:

From the developer page:

>>> from flashtext import KeywordProcessor
>>> keyword_processor = KeywordProcessor()
>>> # keyword_processor.add_keyword(<unclean name>, <standardised name>)
>>> keyword_processor.add_keyword('Big Apple', 'New York')
>>> keyword_processor.add_keyword('Bay Area')
>>> keywords_found = keyword_processor.extract_keywords('I love Big Apple and Bay Area.')
>>> keywords_found
>>> # ['New York', 'Bay Area']

Or when extracting the offset:

>>> from flashtext import KeywordProcessor
>>> keyword_processor = KeywordProcessor()
>>> keyword_processor.add_keyword('Big Apple', 'New York')
>>> keyword_processor.add_keyword('Bay Area')
>>> keywords_found = keyword_processor.extract_keywords('I love big Apple and Bay Area.', span_info=True)
>>> keywords_found
>>> # [('New York', 7, 16), ('Bay Area', 21, 29)]

Limitation: I want to point out that this solution is NOT the optimal solution for the given question. For the given question in from eumiro's solution under the caveat given by @bfontaine in the respective comment is the definetly the best answer.

flashtext is a powerful solution, if you want to find all (!) occurrences of a string in a given text. This is something in cannot do (and is not made for to do).

This is entirely inspired by laurasia's answer above, but it refines the structure.

It also adds some checks:

• It will correctly return 0 when searching an empty file for the empty string. In laurasia's answer, this is an edge case that will return -1.
• It also pre-checks whether the goal string is larger than the buffer size, and raises an error if this is the case.

In practice, the goal string should be much smaller than the buffer for efficiency, and there are more efficient methods of searching if the size of the goal string is very close to the size of the buffer.

def fnd(fname, goal, start=0, bsize=4096):
    if bsize < len(goal):
        raise ValueError("The buffer size must be larger than the string being searched for.")
    with open(fname, 'rb') as f:
        if start > 0:
            f.seek(start)
        overlap = len(goal) - 1
        while True:
            buffer = f.read(bsize)
            pos = buffer.find(goal)
            if pos >= 0:
                return f.tell() - len(buffer) + pos
            if not buffer:
                return -1
            f.seek(f.tell() - overlap)

5000 lines isn't big (well, depends on how long the lines are...)

Anyway: assuming the string will be a word and will be seperated by whitespace...

lines=open(file_path,'r').readlines()
str_wanted="whatever_youre_looking_for"


    for i in range(len(lines)):
        l1=lines.split()
        for p in range(len(l1)):
            if l1[p]==str_wanted:
                #found
                # i is the file line, lines[i] is the full line, etc.