S3 Select Result/Response size

Background:

I recently faced the same question regarding the 1 MB limit. I'm dealing with a large gzip compressed csv file and had to figure out, if S3 Select would be an alternative to processing the file myself. My research makes me feel the author of the previous answer misunderstood the question.

The 1 MB limit referenced by the current AWS S3 Select documentation is referring to the record size:

... The maximum length of a record in the input or result is 1 MB.

The SQL Query is not the input (it has a lower limit though):

... The maximum length of a SQL expression is 256 KB.

Question Response:

I interpret this 1 MB limit the following way:

1. One row in the queried CSV file (uncompressed input) can't use more than 1 MB of memory
2. One result record (result row returned by S3 select) also can't use more than 1 MB of memory

To put this in a practical perspective, the following questions discussed the string size in bytes for Python. I'm using an UTF-8 encoding.

1. This means len(row.encode('utf-8')) (string size in bytes) <= 1024 * 1024 bytes for each csv row represented as UTF-8 encoded string of the input file.
2. It again means len(response_json.encode('utf-8')) <= 1024 * 1024 bytes for each returned response record (in my case the JSON result).

Note:

In my case, the 1 MB limit works fine. However, this depends a lot on the amount of data in your input (and potentially extra, static columns you might add via SQL).

If the limit 1MB is exceeded and you want to query files without a data base solution involved, using the more expensive AWS Athena might be a solution.

Could you point us to part of documentation which talking about this 1mb?

I have never seen 1 MB limit. Downloading of object is just downloading, and you can download almost unlimited file.

AWS Uplaods files with multipart upload and it has limits up to Terabytes for object and up to Gigabytes for objects part

Docs is here: https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

Response to the question

As per comment of author below my post:

Limit described here: https://docs.aws.amazon.com/AmazonS3/latest/dev/querying-glacier-archives.html

This docs refers to query for archived objects. So you can do some query on data, without collecting it from the Glacier.

And input query cannot exceed 1MB. Output of that query cannot exceed 1MB.

• Input is SQL query
• Output is files list.

Find more info here: https://docs.aws.amazon.com/amazonglacier/latest/dev/s3-glacier-select-sql-reference-select.html

So this limit is not for files but for SQL-like queries.