BigQuery + Amazon Athena + Presto: limits on number of partitions and columns

user_6059044 0
Viewed 1758

Based on Google BigQuery documentation, BigQuery has the following limits (https://cloud.google.com/bigquery/quotas):

1) Maximum number of partitions per partitioned table — 4,000.

2) Maximum columns in a table, query result, or view definition — 10,000

Based on Amazon Athena documentation: (https://docs.aws.amazon.com/athena/latest/ug/service-limits.html):

Maximum number of partitions per partitioned table — 20,000

Is there any hard limits on the number of columns for AWS Athena or for Presto?

Thanks!

presto amazon-athena
1 Answers

Is there any hard limits on the number of columns for AWS Athena or for Presto?

Unfortunately I can't answer for AWS Athena, but I can answer for Presto.

No, there are no explicit hard limits for number of columns. In theory, you are limited by available memory and maximum size of collections in Java.

In practice, however, when you have more than 10k columns, you can experience increased planning times. With higher number of columns (perhaps about 100k and more), you will hit JVM byte code length limits in classes generated for the purpose of processing your queries.

user_654580
Related