Is there a way to improve PostgreSQL estimates when joining skewed relations?

Viewed 213

I’m currently working with a PostgreSQL 12 database storing mobile devices. Among others, it has the two tables device_types and device_groups. The data distribution is such that most device_groups have very few device_types in them, e.g. "iOS devices released in 2020" with about 10 device_types, whereas a few device_groups have a very large number of device_types, e.g. "All Android models in existence", with approximately 80'000 device_types.

This is troublesome because the PostgreSQL query planner makes bad estimates and therefore chooses suboptimal join types later in the query when working with the device_group "All Android models in existence". Based on its statistics it expects only very few device_types on average, but in fact has to deal with an intermediate result set of 80'000 devices in some cases.

I have created a minimal reproduction case below.

This is the data model:

CREATE TABLE device_groups (
    id              integer CONSTRAINT device_groups_pk PRIMARY KEY
);

CREATE TABLE device_types (
    id              integer CONSTRAINT device_types_pk PRIMARY KEY,
    device_group_id integer CONSTRAINT device_types_fk_device_groups_id REFERENCES device_groups(id)
);

Now we insert some skewed data. Most device groups have ~10 device_types, one has 80k device_types:

/* create 100 device groups */
INSERT INTO device_groups(id)
SELECT i
FROM generate_series(1, 100) as t(i);

/* 99 device groups have only a few entries, 10 on average: */
INSERT INTO device_types(id, device_group_id)
SELECT i, trunc(random() * 98 + 1)
FROM generate_series(1, 1000) as t(i);

/* device group 100 has 80k devices: */
INSERT INTO device_types(id, device_group_id)
SELECT i, 100
FROM generate_series(1001, 81001) as t(i);

ANALYZE device_groups;
ANALYZE device_types;

This query works well, unsurprisingly, as Postgres has all statistics it needs to estimate the result cardinality:

EXPLAIN ANALYZE
SELECT *
FROM device_types
WHERE device_types.device_group_id = 100;
/* => correctly estimates ~80k rows: */
/* Seq Scan on device_types  (cost=0.00..1371.51 rows=79932 width=8) (actual time=0.104..665.699 rows=80001 loops=1) */

This query (somewhat surprisingly to me) also has the right estimate for how many rows it will return:

EXPLAIN ANALYZE
SELECT *
FROM device_types
JOIN device_groups ON device_types.device_group_id = device_groups.id
WHERE device_groups.id = 100;
/* => also correctly estimates ~80k rows: */
/* Nested Loop  (cost=0.00..2173.08 rows=79932 width=12) (actual time=0.165..1930.591 rows=80001 loops=1) */

Now we add one additional layer of indirection, by adding a third table device_group_collections that we join against, and this is where it gets problematic:

CREATE TABLE device_group_collections (
    device_group_id integer CONSTRAINT device_group_collections_fk_device_groups_id REFERENCES device_groups(id),
    collection_id   integer
);

INSERT INTO device_group_collections(device_group_id, collection_id) VALUES (100, 25);

ANALYZE device_group_collections;

EXPLAIN ANALYZE
SELECT *
FROM device_types
JOIN device_groups ON device_types.device_group_id = device_groups.id
JOIN device_group_collections ON device_group_collections.device_group_id = device_groups.id
WHERE device_group_collections.collection_id = 25;
/* => estimates only 810 rows, but in reality there will be 80'001 rows: */
/* Hash Join  (cost=3.42..1484.29 rows=810 width=20) (actual time=19.817..1949.068 rows=80001 loops=1) */

Is this a general limitation of the Postgres query planner, or is there any way I can help it to understand the skew of the dataset?

1 Answers

This is a limitation of the planner.

Your middle query is accurately estimated only because the planner realizes that ON device_types.device_group_id = device_groups.id WHERE device_groups.id = 100 implies that device_types.device_group_id must also equal 100, and it can look up the stats on device_types.device_group_id for that value.

If your device_groups table also had a "name" column and the query specified ON device_types.device_group_id = device_groups.id WHERE device_groups.name = 'name for 100', then that would not be accurately estimated.

I am not aware of any generic solution to this problem.

Related