I am trying to extract specific lines from a huge text file (over 1.3 GB) with millions of lines. I have tried something like this:
import re
pattern='Blah blah'
my_list=[]
with open(r'C:\Users\Desktop\log.txt') as my_file:
for lines in my_file:
k=re.search(pattern, lines)
if k!=None:
my_list.append(lines)
print(my_list)
The problem is that, spyder breaks and exits before completing the loop. Can you suggest a more memory efficient way to handle this operation?