I have a really large database (billions of rows) that a user can query through a website build with django. I would like to offer the possibility for the user to download the results of their query in excel format. Currently the code is like:
from excel_response import ExcelResponse
qs = data.objects_in(db)
qs_results = qs.filter([...]).order_by([...]])
json_data = [['col1', 'col2', 'col3']]
for item in qs_results:
json_data.append([
escape(item.col1),
escape(item.col2),
escape(item.col3),
])
return ExcelResponse(json_data, 'title')
But this is really slow. The output can sometime have 100,000s of lines (and 20 columns) and take more than 10 minutes to be generated and often times out.
How can I output an excel file with a high number of row efficiently?