How do I process very large file (13GB) in python without crashing?

user_4914597 0
Viewed 642

I have to process this very large file on a server (not my computer). It runs python 64 and has 24 GB of RAM. The file itself is around 13GB in size with 27 million rows of data. Considering the server has pretty big spec, I did try to load the whole to pandas, but it crashed. I tried to use dask, but it's still really slow. So I separated the file in chunks like the following.

My code is similar to this below. I load the file in chunks, where each chunk is 100,000 lines of data. Then it will process each chunk, and append it to an existing file. I thought that by processing things in chunk, it wouldn't store the data in RAM, but I think it still did. The first few hundred iterations worked fine, but sometime after it processed 8GB of data, it just crashed.

chunksize= 100000
c = 0
for chunk in pd.read_csv(fname, chunksize=chunksize,sep='|',error_bad_lines=False):

    chunk['col1'] = chunk['col1'].apply(process1)
    chunk['col2'] = chunk['col2'].apply(process2)

    if c == 0:
        chunk.to_csv("result/result.csv", index=False)
    else:
        chunk.to_csv('result/result.csv', mode='a', header=False, index=False)

    if c%10==0:
        print(c)
        
    c+=1

Usually after 160 iterations which resulted in 8 GB of result.csv file, the program just stopped with a MemoryError:.

I do not have access to many things in this sever to be honest, so if you want to suggest changing some settings that I don't have access to, then I probably cannot. But let's see what I can do. Thanks in advance.

EDIT: I will add what's going on in process1 and process2 here.

def process1(name):
    if type(name)==str:
        new_name = name[:3]+'*' * len(name[:-3])
    else:
        return name
    
    return new_name

def process2(number):
    if number !=np.nan:
        new_number = str(number)
        new_number = '*'*len(new_number)
        return new_number
    else:
        return number
pandas dataframe memory memory-management python
1 Answers

The general syntax of a for loop is

for target in expression:
    do all the things

Python will evaluate the expression to an object and only when that is complete, it will assign the object to the target variable. That means that any object already in target is not deleted until after its replacement has been built.

That's not a big deal unless the object being created is large. That's the case here. The about-to-be deleleted chunk is in memory when the new one is created, effectively doubling the impact on memory. The solution is to manually delete the target in the loop before returning for more.

for chunk in pd.read_csv(fname, chunksize=chunksize,sep='|',error_bad_lines=False):

    chunk['col1'] = chunk['col1'].apply(process1)
    chunk['col2'] = chunk['col2'].apply(process2)

    if c == 0:
        chunk.to_csv("result/result.csv", index=False)
    else:
        chunk.to_csv('result/result.csv', mode='a', header=False, index=False)
    del chunk # destroy dataframe before next loop to conserve memory.    
    if c%10==0:
        print(c)
    c+=1
user_6420700
Related