I'm trying process a file based on number of lines. After 10 lines, it will trigger threading. The reason for this; I'm reading a big file that contains 1 Million+ lines, it's not convenient to put all of them into a queue. I'm using itertools.islice to easily read n lines and then process them after reaching 10 lines.
from itertools import islice
from threading import Thread
from queue import Queue
num = 10 #Number of Threads & Lines / queue
qlist = Queue()
...
with open('target.txt') as f:
for line in f:
liner = [line] + list(islice(f, num-1))
for i in liner:
qlist.put(i)
inner = Thread(target=do_work)
After 30+ minute run, it raise an exception too many open files; I assume this is because of the files are not closing after read n lines. The reason i put the Thread inside the open files is to process the queue right away, while it has n task. I don't want to raise ulimit or number of openfiles limit. An ugly way to solve this:
from time import sleep
control = {'total':0,'line':0}
append = []
target = './1MillWordlist'
def ends():
print('You finally finish')
sleep(3)
exit()
def bla():
with open(target) as f:
for i, line in enumerate(f):
if i == switch['line']:
append.append(line)
for i in append:
print(i)
switch['line']+=1
if not control['line'] == control['total']:
bla()
else:
return
with open(target) as f:
for i, line in enumerate(f):
pass
control['total']=i
bla()
ends()
Is there an appropiate way to close the file after the n lines already in queue?