I’m currently working with a PostgreSQL 12 database storing mobile devices. Among others, it has the two tables device_types and device_groups. The data distribution is such that most device_groups have very few device_types in them, e.g. "iOS devices released in 2020" with about 10 device_types, whereas a few device_groups have a very large number of device_types, e.g. "All Android models in existence", with approximately 80'000 device_types.
This is troublesome because the PostgreSQL query planner makes bad estimates and therefore chooses suboptimal join types later in the query when working with the device_group "All Android models in existence". Based on its statistics it expects only very few device_types on average, but in fact has to deal with an intermediate result set of 80'000 devices in some cases.
I have created a minimal reproduction case below.
This is the data model:
CREATE TABLE device_groups (
id integer CONSTRAINT device_groups_pk PRIMARY KEY
);
CREATE TABLE device_types (
id integer CONSTRAINT device_types_pk PRIMARY KEY,
device_group_id integer CONSTRAINT device_types_fk_device_groups_id REFERENCES device_groups(id)
);
Now we insert some skewed data. Most device groups have ~10 device_types, one has 80k device_types:
/* create 100 device groups */
INSERT INTO device_groups(id)
SELECT i
FROM generate_series(1, 100) as t(i);
/* 99 device groups have only a few entries, 10 on average: */
INSERT INTO device_types(id, device_group_id)
SELECT i, trunc(random() * 98 + 1)
FROM generate_series(1, 1000) as t(i);
/* device group 100 has 80k devices: */
INSERT INTO device_types(id, device_group_id)
SELECT i, 100
FROM generate_series(1001, 81001) as t(i);
ANALYZE device_groups;
ANALYZE device_types;
This query works well, unsurprisingly, as Postgres has all statistics it needs to estimate the result cardinality:
EXPLAIN ANALYZE
SELECT *
FROM device_types
WHERE device_types.device_group_id = 100;
/* => correctly estimates ~80k rows: */
/* Seq Scan on device_types (cost=0.00..1371.51 rows=79932 width=8) (actual time=0.104..665.699 rows=80001 loops=1) */
This query (somewhat surprisingly to me) also has the right estimate for how many rows it will return:
EXPLAIN ANALYZE
SELECT *
FROM device_types
JOIN device_groups ON device_types.device_group_id = device_groups.id
WHERE device_groups.id = 100;
/* => also correctly estimates ~80k rows: */
/* Nested Loop (cost=0.00..2173.08 rows=79932 width=12) (actual time=0.165..1930.591 rows=80001 loops=1) */
Now we add one additional layer of indirection, by adding a third table device_group_collections that we join against, and this is where it gets problematic:
CREATE TABLE device_group_collections (
device_group_id integer CONSTRAINT device_group_collections_fk_device_groups_id REFERENCES device_groups(id),
collection_id integer
);
INSERT INTO device_group_collections(device_group_id, collection_id) VALUES (100, 25);
ANALYZE device_group_collections;
EXPLAIN ANALYZE
SELECT *
FROM device_types
JOIN device_groups ON device_types.device_group_id = device_groups.id
JOIN device_group_collections ON device_group_collections.device_group_id = device_groups.id
WHERE device_group_collections.collection_id = 25;
/* => estimates only 810 rows, but in reality there will be 80'001 rows: */
/* Hash Join (cost=3.42..1484.29 rows=810 width=20) (actual time=19.817..1949.068 rows=80001 loops=1) */
Is this a general limitation of the Postgres query planner, or is there any way I can help it to understand the skew of the dataset?