Postgres JSONB Retrieval is very slow

user_1995391 0
Viewed 1889

At bit of a loss here. First and foremost, I'm not a dba nor do I really have any experience with Postgres with the exception of what I'm doing now.

Postgres seems to choke when you want to return jsonb documents in general for anything more than a couple of hundred rows. When you try to return thousands, query performance becomes terrible. If you go even further and attempt to return multiple jsonb documents following various table joins, forget it.

Here is my scenario with some code:

I have 3 tables - all tables have jsonb models, all complex models and 2 of which are sizeable (8 to 12kb in size uncompressed). In this particular operation I need to unnest a jsonb array of elements to then work through - this gives me roughly 12k records.

Each record then contains an ID that I use to join another important table - I need to retreive the jsonb doc from this table. From there, I need to join that table on to another (much smaller) table and also pull the doc from there based on another key.

The output is therefore several columns + 3 jsonb documents ranging from <1kb in size to around 12kb uncompressed in size.

Query data retrieval is effectively pointless - I've yet to see the query return data. As soon as I strip away the json doc columns, naturally the query speeds up to seconds or less. 1 jsonb document bumps the retrieval to 40seconds in my case, adding a second takes us to 2 minutes and adding the third is much longer.

What am I doing wrong? Is there any way to retrieve the jsonb documents in a performant way?

SELECT x.id,
    a.doc1,
    b.doc2,
    c.doc3
   FROM ( SELECT id,
            (elements.elem ->> 'a'::text)::integer AS a,
            (elements.elem ->> 'b'::text)::integer AS b,
            (elements.elem ->> 'c'::text)::integer AS c,
            (elements.elem ->> 'd'::text)::integer AS d,
            (elements.elem ->> 'e'::text)::integer AS e
           FROM tab
             CROSS JOIN LATERAL jsonb_array_elements(tab.doc -> 'arrayList'::text) WITH ORDINALITY elements(elem, ordinality)) x
     LEFT JOIN table2 a ON x.id = a.id
     LEFT JOIN table3 b ON a.other_id = b.id
     LEFT JOIN table4 c ON b.other_id = c.id;

The tables themselves are fairly standard:

CREATE TABLE a ( 
  id (primary key), 
  other_id (foreign key), 
  doc jsonb 
)

Nothing special about these tables, they are ids and jsonb documents

A note - we are using Postgres for a few reasons, we do need the relational aspects of PG but at the same time we need to document storage and retrieval ability for later in our workflow.

Apologies if I've not provided enough data here, I can try to add some more based on any comments

EDIT: added explain:

                                                    QUERY PLAN
------------------------------------------------------------------------------------------------------------------
 Hash Left Join  (cost=465.79..96225.93 rows=11300 width=1843)
   Hash Cond: (pr.table_3_id = br.id)
   ->  Hash Left Join  (cost=451.25..95756.86 rows=11300 width=1149)
         Hash Cond: (((p.doc ->> 'secondary_id'::text))::integer = pr.id)
         ->  Nested Loop Left Join  (cost=0.44..95272.14 rows=11300 width=1029)
               ->  Nested Loop  (cost=0.01..239.13 rows=11300 width=40)
                     ->  Seq Scan on table_3  (cost=0.00..13.13 rows=113 width=710)
                     ->  Function Scan on jsonb_array_elements elements  (cost=0.01..1.00 rows=100 width=32)
               ->  Index Scan using table_1_pkey on table_1 p  (cost=0.43..8.41 rows=1 width=993)
                     Index Cond: (((elements.elem ->> 'field_id'::text))::integer = id)
         ->  Hash  (cost=325.36..325.36 rows=10036 width=124)
               ->  Seq Scan on table_2 pr  (cost=0.00..325.36 rows=10036 width=124)
   ->  Hash  (cost=13.13..13.13 rows=113 width=710)
         ->  Seq Scan on table_3 br  (cost=0.00..13.13 rows=113 width=710)
(14 rows)

EDIT2: Sorry been mega busy - I will try to go into more detail - firstly the fully explain plan (I didn't know about the additional parameters) - Ill leave in the actual tables (I wasn't sure if I was allowed to):

Hash Left Join  (cost=465.79..96225.93 rows=11300 width=1726) (actual time=4.669..278.781 rows=12522 loops=1)
   Hash Cond: (pr.brand_id = br.id)
   Buffers: shared hit=64813
   ->  Hash Left Join  (cost=451.25..95756.86 rows=11300 width=1032) (actual time=4.537..265.749 rows=12522 loops=1)
         Hash Cond: (((p.doc ->> 'productId'::text))::integer = pr.id)
         Buffers: shared hit=64801
         ->  Nested Loop Left Join  (cost=0.44..95272.14 rows=11300 width=912) (actual time=0.240..39.480 rows=12522 loops=1)
               Buffers: shared hit=49964
               ->  Nested Loop  (cost=0.01..239.13 rows=11300 width=40) (actual time=0.230..8.177 rows=12522 loops=1)
                     Buffers: shared hit=163
                     ->  Seq Scan on brand  (cost=0.00..13.13 rows=113 width=710) (actual time=0.003..0.038 rows=113 loops=1)
                           Buffers: shared hit=12
                     ->  Function Scan on jsonb_array_elements elements  (cost=0.01..1.00 rows=100 width=32) (actual time=0.045..0.057 rows=111 loops=113)
                           Buffers: shared hit=151
               ->  Index Scan using product_variant_pkey on product_variant p  (cost=0.43..8.41 rows=1 width=876) (actual time=0.002..0.002 rows=1 loops=12522)
                     Index Cond: (((elements.elem ->> 'productVariantId'::text))::integer = id)
                     Buffers: shared hit=49801
         ->  Hash  (cost=325.36..325.36 rows=10036 width=124) (actual time=4.174..4.174 rows=10036 loops=1)
               Buckets: 16384  Batches: 1  Memory Usage: 1684kB
               Buffers: shared hit=225
               ->  Seq Scan on product pr  (cost=0.00..325.36 rows=10036 width=124) (actual time=0.003..1.836 rows=10036 loops=1)
                     Buffers: shared hit=225
   ->  Hash  (cost=13.13..13.13 rows=113 width=710) (actual time=0.114..0.114 rows=113 loops=1)
         Buckets: 1024  Batches: 1  Memory Usage: 90kB
         Buffers: shared hit=12
         ->  Seq Scan on brand br  (cost=0.00..13.13 rows=113 width=710) (actual time=0.003..0.043 rows=113 loops=1)
               Buffers: shared hit=12
 Planning Time: 0.731 ms
 Execution Time: 279.952 ms
(29 rows)
jsonb query-performance database-performance postgresql
1 Answers

Your query is hard to follow for a few reasons:

  1. Your tables are named tab, table2, table3, `table4.
  2. Your subquery parses JSON for every single row in the table, projects out some values, and then the outer query never uses those values. The only one that appears to be relevant is id.
  3. Outer joins must be executed in order while inner joins can be freely re-arranged for performance. Without knowing the purpose of this query, it's impossible for me to determine if an outer join is appropriate or not.
  4. The table names and column names in the execution plan do not match the query, so I'm not convinced this plan is accurate.
  5. You did not supply a schema.

That said, I'll do my best.

Things that stand out performance-wise

No where clause

Since there is no where clause, your query will run jsonb_array_elements against every single row of tab, which is what is happening. Aside from extracting the data out of JSON and storing it into a separate column, I can't imagine much that could be done to optimize it.

Insufficient indexes to power the joins

The query plan suggests there might be a meaningful cost to the joins. Except for table1, each join is driven by a sequential scan of the tables, which means, reading every row of both tables. I suspect adding indexes on each table will help. It appears you are joining on id columns, so a simple primary key constraint will improve both your data integrity and query performance.

alter table tab add constraint primary key (id);
alter table table2 add constraint primary key (id);
alter table table3 add constraint primary key (id);
alter table table4 add constraint primary key (id);

Type conversions

This part of the execution plan shows a double type conversion in your first join:

Index Cond: (((elements.elem ->> 'field_id'::text))::integer = id)

This predicate means that the id value from tab is being converted to text, then the text converted to an integer so it can match against table2.id. These conversions can be expensive in compute time and it some cases, can prevent index usage. It's hard to advise on what to do because I don't know what the actual types are.

user_16491980
Related