How to efficiently export large Excel file with python and Django

Viewed 1335

I have a really large database (billions of rows) that a user can query through a website build with django. I would like to offer the possibility for the user to download the results of their query in excel format. Currently the code is like:

from excel_response import ExcelResponse

qs = data.objects_in(db)
qs_results = qs.filter([...]).order_by([...]])

json_data = [['col1', 'col2', 'col3']]
    
    for item in qs_results:
    
    json_data.append([
        escape(item.col1),
        escape(item.col2),
        escape(item.col3),
    ])

    return ExcelResponse(json_data, 'title')

But this is really slow. The output can sometime have 100,000s of lines (and 20 columns) and take more than 10 minutes to be generated and often times out.

How can I output an excel file with a high number of row efficiently?

1 Answers
Related