Get data out of a file without iterating through it multiple times

Viewed 87

I've created the following function to pull data out of a file. It works ok, but gets very slow for larger files.

def get_data(file, indexes, data_start, sieve_first = is_float):
    file_list = list(file)
    for i in indexes:
        d_line = i+data_start
        for line in file_list[d_line:]:
            if sieve_first(line.strip().split(',')[0]):
                yield file_list[d_line].strip()
                d_line += 1
            else:
                break

def is_float(f):
    try:
        float(str(f))
    except:
        return False
    else:
        return True

with open('my_data') as f:
    data = get_data(f, index_list, 3)

The file might look like this (line numbers added for clarity):

line 1234567: # <-- INDEX
line 1234568: # +1
line 1234569: # +2
line 1234570:      8, 17.0, 23, 6487.6
line 1234571:      8, 17.0, 23, 6487.6
line 1234572:      8, 17.0, 23, 6487.6
line 1234572:
line 1234572:
line 1234572:

With the above example, lines 1234570 through 1234572 will be yielded.

Since my files are large, there are a couple things I don't like about my function.

  1. First is that it reads the entire file into memory; I do this so I can use line indexing in order to parse the data out.
  2. Second is that the same lines in the file are iterated over many times- this gets very expensive for a large file.

I have fiddled around trying to use iterators to get through the file a single time, but haven't been able to crack it. Any suggestions?

2 Answers
Related