Reading large number of files(20k+) in python memory Error

user_9943196 0
Viewed 392

I am trying to read a large number of files 20k+ from my computer using python but I keep on getting this Memory ERROR(details below). Although I have 16GB of RAM from which 8 GB or more is free all the time and the size of all the files is just 270Mb. I have tried many different solutions like pandas read_csv() reading in chunks using open(file_path).read(100) and many others but I am unable to read the files. I have to create a corpus of words after reading the files in the list. Below is my code so far. Any help will be highly appreciated.

import os
import pandas as pd

collectionPath = r"C:\Users\Asghar Nazir\OneDrive - Higher Education Commission\MSDS\S1\IR\assignment\ACL txt"

listOfFilesInCollection = os.listdir(collectionPath)

def wordList(file):
    list_of_words_from_file =[]
    for line in file:
        for word in line.split():
            list_of_words_from_file.append(word)
    return list_of_words_from_file


list_of_file_word_Lists = {}
file=[]
for file_name in listOfFilesInCollection:
    filePath = collectionPath + "\\" + file_name
    with open(filePath) as f:
        for line in f:
            file.append(line)
    list_of_file_word_Lists[file_name]=wordList(file)
print(list_of_file_word_Lists)

The error that I get

Traceback (most recent call last): File "C:/Users/Asghar Nazir/PycharmProjects/pythonProject/IRAssignment1/init.py", line 25, in list_of_file_word_Lists[file_name]=wordList(file) File "C:/Users/Asghar Nazir/PycharmProjects/pythonProject/IRAssignment1/init.py", line 14, in wordList list_of_words_from_file.append(word) MemoryError

python-3.x file
2 Answers

You probably want to move the file=[] at the beginning of the loop because you're currently adding lines of each new file you open without having first removed the lines of all the previous files.

Then, there're very likely more efficient approaches depending on what you're trying to achieve. If the order of words doesn't matter, then maybe using using a dict or a collections.Counter instead of a list can help to avoid duplication of identical strings. If neither the order nor the frequency of words matter, then maybe using a set instead is going to be even better.

Finally, since it's likely you'll find most words in several files, try to store each of them only once in memory. That way, you'll be able to scale way higher than a mere 20k files: there's plenty of space in 16 GiB of RAM.

Keep in mind that Python has lots of fixed overheads and hidden costs: inefficient data structures can cost way more than you would expect.

user_28668980

It is hard to tell why your memory problems arise without knowing the content of your files. Maybe it is enough to make your code more efficient. For example: The split()-function can handle multiple lines itself. So you don't need a loop for that. And using list comprehension is always a good idea in python.

The following code should return what you want and I don't see a reason why you should run out of memory using it. Besides that, Arkanosis' hint to the importance of data types is very valid. It depends on what you want to achieve with all those words.

from pathlib import Path

def word_list_from_file(path):
    with open(path, 'rt') as f:
        list_words = f.read().split()
    return list_words
    
path_dir = Path(r"C:\Users\Asghar Nazir\OneDrive - Higher Education Commission\MSDS\S1\IR\assignment\ACL txt")
dict_file_content = {
    str(path.name): word_list_from_file(path)
    for path in path_dir.rglob("*.txt")
}

P.S.: I'm not sure how the pathlib-module works in windows. But from what I read, this code snippet is platform independent.

user_136825590
Related