How to send a large file over HTTP response in Google Cloud Function

user_15042433 0
Viewed 1093

I've tried the code below, but Google Cloud Function's limit for a response is only 10M, but I want to return larger files:

const csv = json2csv(onlyDataTransactions);
response.setHeader(
  "Content-disposition",
  "attachment; filename=transactions.csv"
);
response.set("Content-Type", "text/csv");
response.status(200).send(csv);

Updated: Thanks to @Andrew I have this first update on code, I force the compression because compression middleware on firebase cloud functions depends even on user-agent header, I still working o the other suggestions to find the best result, thanks to all.

if (request.headers['content-type'] === 'text/csv') {
    const onlyDataTransactions = transactions.map(transaction => transaction.toCsvRecord());
    const csv = parse(onlyDataTransactions);
    response.setHeader(
       "Content-disposition",
       "attachment; filename=transactions.csv"
    );
    response.set("Content-Type", "text/csv");
    response.set('Content-Encoding', 'gzip');
    const content = await gzip(JSON.stringify(csv));
    response.status(200).send(content);
}
google-cloud-functions node.js
3 Answers

How large is the .csv that you are transferring?

Documentation:

Note: Cloud Functions limits HTTP request body sizes to 10MB, so any request larger than this will be rejected before your function is executed. We recommend uploading large files or files that persist beyond a single request directly to Cloud Storage

Based on this, I would introduce some compression of the .csv by adding Content-Encoding to convert files to and from a gzip-compressed state. You should get a good rate of compression if it is a .csv file.

Content-Type: text/plain
Content-Encoding: gzip

If you are using Google Cloud Storage (GCS), you can read more about transcoding in the GCS documentation.

gzip is a form of data compression: it typically reduces the size of a file. This allows the file to be transferred faster and stored using less space than if it were not compressed. Compressing a file can reduce both cost and transfer time. Transcoding, in Cloud Storage, is the automatic changing of a file's compression before it's served to a requester. When transcoding results in a file becoming gzip-compressed, it can be considered compressive, whereas when the result is a file that is no longer gzip-compressed, it can be considered decompressive. Cloud Storage supports the decompressive form of transcoding.

There is also an excellent resource on the Mozilla site that is worth reading about setting the header to compress and decompress media.

If this does not work for you, then your only real option is to either chunk the data somehow or compress the file while it is still in the GCS bucket. I have had success with gzip compression levels in the past to reduce the file size. The compression could be done with another Cloud Function if needed.

user_116301380
user_55257440
user_113725930
Related