Improving TS-vector Query with JOIN statements

Viewed 39

Hi all nice people out there.
on Postgres (Version 9.6) I have 3 tables

  1. app_complaint_main (8 Million records)
PK company_id consumer_complaint_narrative_id complaint_id
1 33 44 55
  1. app_company (6 K records)
PK company_name
33 'ACME'
  1. app_consumer_complaint_narrative (2.5 Million records)
PK text fts_document(tsvector)
44 "Hello World" lexems

the text search is absolutely amazing - it went from 7 Min. to 100 MS,
however when it comes to cases where I need to find the companies that have complaints with the desired text + a list of the complaint_ids that contained this word it got messy... because it must loop through the main table for both company_id and complaint_id relations (and yes, I did use an Index for the consumer_complaint_narrative_id forign_key)

Disclaimer - I recently found out that the app_consumer_complaint_narrative table contains duplicate texts which, in turn, caused duplicate lexems. but when I tried to use DISTINCT on the search it took a long time(the GIN Index is not designed for comparing a TSvector filed to another).
so I'm facing 2 issues:

  • duplicate lexems to run on

  • 2 join statements to workaround attached is the Query I'm currently using

     SELECT a.company_name ,a.ids
    from(
    SELECT ac.company_name, ARRAY_AGG(distinct  acm.complaint_id) ids
    FROM public.app_complaint_main acm
     join public.app_company ac 
    on acm.company_id = ac.id
    join public.app_consumer_complaint_narrative accn
    on acm.consumer_complaint_narrative_id = accn.id
     WHERE fts_document @@ to_tsquery('english','fish')
     group by ac.company_name
    )a 
    
    

analysis :
# Node Timings Rows Loops Exclusive Inclusive Rows X Actual Plan

  1. Aggregate (cost=83408.52..87419.85 rows=6726 width=59) (actual=104.14..107.721 rows=163 loops=1) 0.344 ms 107.721 ms ↑ 41.27 163 6726 1
  2. Gather Merge (cost=83408.52..87174.12 rows=32332 width=31) (actual=104.113..107.378 rows=756 loops=1) -113.434 ms 107.378 ms ↑ 42.77 756 32332 1
  3. Sort (cost=82408.5..82442.18 rows=13472 width=31) (actual=73.585..73.604 rows=252 loops=3) 1.479 ms 220.812 ms ↑ 53.47 252 13472 3
  4. Hash Inner Join (cost=355.79..81484.48 rows=13472 width=31) (actual=5.113..73.111 rows=252 loops=3) Hash Cond: (acm.company_id = ac.id) 1.625 ms 219.334 ms ↑ 53.47 252 13472 3
  5. Nested Loop Inner Join (cost=134.46..81227.76 rows=13472 width=8) (actual=1.314..68.908 rows=252 loops=3) 1.648 ms 206.724 ms ↑ 53.47 252 13472 3
  6. Bitmap Heap Scan on app_consumer_complaint_narrative as accn (cost=134.02..45406.89 rows=5270 width=4) (actual=1.265..45.282 rows=268 loops=3) Recheck Cond: (fts_document @@ '''fish'''::tsquery) Heap Blocks: exact=342 132.793 ms 135.846 ms ↑ 19.67 268 5270 3
  7. Bitmap Index Scan using fts_idx (cost=0..130.86 rows=12648 width=0) (actual=3.053..3.053 rows=805 loops=1) Index Cond: (fts_document @@ '''fish'''::tsquery) 3.053 ms 3.053 ms ↑ 15.72 805 12648 1
  8. Index Scan using app_complaint_main_consumer_complaint_narrative_id_idx on app_complaint_main as acm (cost=0.43..6.79 rows=1 width=12) (actual=0.085..0.086 rows=1 loops=805) Index Cond: (consumer_complaint_narrative_id = accn.id) 69.23 ms 69.23 ms ↑ 1 1 1 805
  9. Hash (cost=137.26..137.26 rows=6726 width=31) (actual=3.662..3.662 rows=6736 loops=3) Buckets: 8192 Batches: 1 Memory Usage: 489 kB 5.248 ms 10.986 ms ↓ 1.01 6736 6726 3
  10. Seq Scan on app_company as ac (cost=0..137.26 rows=6726 width=31) (actual=0.02..1.913 rows=6736 loops=3) 5.739 ms 5.739 ms ↓ 1.01 6736 6726 3
0 Answers
Related