I need to sort the data based on an integer ID value in the first column in the .csv file. I used to sort my .csv data using the following script. However, my data is much bigger now and contains billions of records and I cannot load it in the memory to use the sorted method in python. Any ideas or suggestion is appreciated.
ID_idx = 0
data=[]
with open(file_path) as data_file:
header=next(data_file)
for line in data_file:
line=line.split(',')
data.append(line)
data_sorted = sorted(data,key=itemgetter(ID_idx), reverse=False)
with open('data_sorted.csv', 'w') as data_file:
data_file.write(','.join(header.split(',')))
data_file.write('\n')
for i in range(len(data_sorted)):
data_file.write(','.join(map(repr, data_sorted[i])))
data_file.write('\n')