Need Information on Snowflake optimizer

I find "knowing the 'rules'" less helpful, than understanding what the system is doing as more helpful.

I have found describing it to new team members has massive table scans, that do map/reduce/merge joins.

You can make the tables scans faster by selecting the smallest set of columns needed to get the answer you need.

There is partition pruning so if you have data in a 'inserted/sorted' order of x 1-2,3-4,5-6 and your query has x = 5, then the first two partitions will not be read.

Next because it's all merge joins, equi joins are the fastest thing todo. [Edit:] This is trying to say, that at the order of million's of rows and up. Joining 1m rows to 1m rows based on complex join logic like a.v1 > b.v2 or a.v2 < b.v3 ... etc means you have to more or less make you trillion+ rows and just try and see. Where-as if you can join on exact values a.v1 = b.v2 and a.v2 = b.v2 now the data can be sorted with respect to those keys, and a merge join can be done, and your performance is very good (sort-merge join on Wikipedia).

This means sometimes reading from the same set of source tables many times in different CTE's and joining those can be the fastest way to process large volumes of data. [Edit:] which in the context of the above statement often in small db SQL people do correlated sub-queries, because a) you can, so why not, and b) they can be fast on indexed databases. But in snowflake with no indexes, besides the optimizer doesn't support most correlated sub-queries, you generally should avoid them and read the data twice in two CTEs and join/left-join those via a equi-join to answer the question that is done, as the CTE's tasks are independent, thus parallelisable, and the merge-join is near-optimal. And the waste of calculating (lets pretend sub-totals) for data that is not in the main join body, is less that gains of parallelism. (this holds best for queries in the 30 seconds or longer range, as compared to speeding up sub 5 second sized queries). But with everything, have a base model, and try/experiment, and poke and the slow stuff, till you cannot restructure you data or query to make it faster.

As always look at the profile of the run query, and look for area there many rows are dropped, and think how you can restructure the logic to push these restrictions earlier in the pipeline.

Brief description could be found in the following document: The Snowflake Elastic Data Warehouse by Snowflake Computing

3.3.1 Query Management and Optimization

(...)

Snowflake’s query optimizer follows a typical Cascades- style approach [28], with top-down cost-based optimization. All statistics used for optimization are automatically main- tained on data load and updates. Since Snowflake does not use indices (cf. Section 3.3.3), the plan search space is smaller than in some other systems. The plan space is further reduced by postponing many decisions until execu- tion time, for example the type of data distribution for joins. This design reduces the number of bad decisions made by the optimizer, increasing robustness at the cost of a small loss in peak performance. It also makes the system easier to use (performance becomes more predictable), which is in line with Snowflake’s overall focus on service experience. Once the optimizer completes, the resulting execution plan is distributed to all the worker nodes that are part of the query. As the query executes, Cloud Services continuously tracks the state of the query to collect performance counters and detect node failures. All query information and statis- tics are stored for audits and performance analysis. (...)

Query Optimization:

Snowflake supports query vectorization and does some cost-based optimization. But its first-time run of queries are typically seconds-to-minutes. Snowflake has added local disk “caching” and also a result cache to speed up subsequent queries for repetitive workloads like reporting and dashboards.

Optimized Storage:

Snowflake has a micro-partition file system that is more optimized than S3 and supports partitioning and sorting with cluster keys Split the data into multiple small files to support optimal data loading in Snowflake.

IMPROVING QUERY PERFORMANCE

1. Consider implementing clustering keys for large tables.
2. Try to execute relatively homogeneous queries (size, complexity, data sets, etc.) on the same warehouse.

IMPROVING LOAD PERFORMANCE

1. Use bulk loading to get the data into tables in Snowflake. Consider splitting large data files so the load can be efficiently distributed across servers in a cluster.
2. Delete from internal stages files that are no longer needed. You may notice an improvement performance in addition to saving on costs.
3. Isolate load and transform jobs from queries to prevent resource contention. Dedicate separate warehouses for loading and querying operations to optimize performance for each.
4. Leverage the scalable compute layer to do the bulk of the data processing. Consider using Snowpipe in micro-batching scenarios. Your query may benefit from cached results from a previous execution.
5. Use separate warehouses for your queries and load tasks. This will facilitate targeted provisioning of warehouses and avoid any resource contention between dissimilar operations.
6. Use a separate data warehouse for large files.
   The number and capacity of the servers determine the number of data files.

Segment Data

Snowflake caches data in the virtual data warehouse, but it's still essential to segment data. Consider these best practices for data query performance:

Group users with common queries in the same virtual data warehouse to optimize data retrieval and use.
The Snowflake Query Profile supports query analysis to help identify and address performance concerns. Snowflake draws from the same virtual data warehouse to support complex data science operations, business intelligence queries, and ELT data integration.

Scale-Up

Snowflake allows for a scale-up in the virtual data warehouse to better handle large workloads. When using scale-up to improve performance, make note of the following:

Snowflake supports fast and easy adjustments to the warehouse-size to handle the workload.
It can also automatically suspend or resume the scale-up, with complete transparency for the user.
Snowflake's scale-up functionality supports the continually changing requirements for processing.

Scale-Out

Snowflake supports the deployment of same-size clusters to support concurrency. Keep these points in mind for how scale-out can help performance optimization:

As users execute queries, the virtual data warehouse automatically adds clusters up to a fixed limit.
It can scale-up in a more controlled way instead of deploying one or more clusters of larger machines like legacy data platforms.
Snowflake automatically adjusts based on user queries, with automatic clustering during peak and off hours as needed.