Saving the content of large .txt file into variable in python

user_7079215 0
Viewed 173

I have a huge .txt file containing approx 100m lines, and I want to save its content into a list line by line.

I'm using the following code to achive this:

tmp=[]
with open('xy.txt') as f:
    for line in f:
        tmp.append(line)

Now the problem with this, that this code doesn't release the memory until the whole file is read, thus after a while it runs out of memory.

My question is, that is there any way of completing this task memory efficiently?

append python-3.x
1 Answers

You can define a generator that yields line by line:

def gen():
    with open('xy.txt') as fh:
        yield from fh

If you need only a specific part of it, you can use itertools.islice to select that slice:

import itertools as it

data = it.islice(gen(), 10**6, 10**7)
user_37672390
Related