How to sort very large .csv files including billions of records in python?

Viewed 226

I need to sort the data based on an integer ID value in the first column in the .csv file. I used to sort my .csv data using the following script. However, my data is much bigger now and contains billions of records and I cannot load it in the memory to use the sorted method in python. Any ideas or suggestion is appreciated.

ID_idx = 0
data=[]
with open(file_path) as data_file:
  header=next(data_file)  
  for line in data_file:
    line=line.split(',')
    data.append(line)
data_sorted = sorted(data,key=itemgetter(ID_idx), reverse=False)
with open('data_sorted.csv', 'w') as data_file:
    data_file.write(','.join(header.split(',')))
    data_file.write('\n')
    for i in range(len(data_sorted)):
        data_file.write(','.join(map(repr, data_sorted[i])))
        data_file.write('\n')

0 Answers
Related