I've created the following function to pull data out of a file. It works ok, but gets very slow for larger files.
def get_data(file, indexes, data_start, sieve_first = is_float):
file_list = list(file)
for i in indexes:
d_line = i+data_start
for line in file_list[d_line:]:
if sieve_first(line.strip().split(',')[0]):
yield file_list[d_line].strip()
d_line += 1
else:
break
def is_float(f):
try:
float(str(f))
except:
return False
else:
return True
with open('my_data') as f:
data = get_data(f, index_list, 3)
The file might look like this (line numbers added for clarity):
line 1234567: # <-- INDEX
line 1234568: # +1
line 1234569: # +2
line 1234570: 8, 17.0, 23, 6487.6
line 1234571: 8, 17.0, 23, 6487.6
line 1234572: 8, 17.0, 23, 6487.6
line 1234572:
line 1234572:
line 1234572:
With the above example, lines 1234570 through 1234572 will be yielded.
Since my files are large, there are a couple things I don't like about my function.
- First is that it reads the entire file into memory; I do this so I can use line indexing in order to parse the data out.
- Second is that the same lines in the file are iterated over many times- this gets very expensive for a large file.
I have fiddled around trying to use iterators to get through the file a single time, but haven't been able to crack it. Any suggestions?