how to fast order result subquery in mysql?

Viewed 33

I have Query

select *
from (select p.id,
         (select sum(price_discount * order_count)
          from product_log
          where product_id = p.id
            and create_at >= 1660590000
            and create_at <= 1663268400) revenue
  from product p
  where p.id in (select product_id from category_product where category_id = 8712)) temp
order by revenue desc

there are 2 million rows in the subquery TEMP, which is calculated in 1.3 seconds. but sorting this result is very long, almost 3 minutes. how to speed it up?

category and product relationship:

create table category_product
(
    category_id int null,
    product_id  int null,
    constraint i_category_product
        unique (category_id, product_id),
    constraint i_product_category
        unique (product_id, category_id)
);

product

create table product
(
    id        int auto_increment
        primary key,
    create_at int                  null comment 'Создан',
    update_at int                  null comment 'Обновлен',
    status    tinyint(1) default 1 null comment 'Статус',
    group_id  int                  null comment 'Группа',
    pics      int                  null comment 'кол-во изображений',
    is_new    tinyint(1) default 0 null comment 'Новинка',
    lost_sale int                  null comment 'упущеная',
    last_log  smallint             null comment 'Дата последнего лога'
);

create index `idx-product_copy-group_id`
    on product (group_id);

product logs

create table product_log
(
    create_at      int            not null comment 'Создан',
    product_id     int            not null,
    position       int            null comment 'Позиция',
    price          int            not null,
    price_discount decimal(10, 2) null comment 'Цена со скидкой',
    price_client   decimal(10, 2) null comment 'Цена со скидкой постоянного покупателя',
    rating         int            null comment 'Рейтинг',
    comment        int            null comment 'Кол-во комментариев',
    order_count    int            null comment 'Кол-во продаж',
    sales_count    int            null comment 'Кол-во продаж',
    refund_count   int            null comment 'Кол-во возвратов/поступлений',
    quantity       int            null comment 'Кол-во на складе',
    lost_sale      decimal(10, 2) null comment 'упущеная',
    constraint `ix-product_log-product_id-category_id`
        unique (product_id, create_at)
);

create index `idx-product_log-product_id`
    on product_log (product_id);

create index `ix-order-refund`
    on product_log (order_count, refund_count);

result of explain analyze

-> Nested loop inner join  (cost=515368.96 rows=1140550) (actual time=0.017..991.153 rows=592913 loops=1)
    -> Index lookup on category_product using i_category_product (category_id=8712)  (cost=116176.46 rows=1140550) (actual time=0.008..267.923 rows=592913 loops=1)
    -> Single-row index lookup on p using PRIMARY (id=category_product.product_id)  (cost=0.25 rows=1) (actual time=0.001..0.001 rows=1 loops=592913)
-> Select #2 (subquery in projection; dependent)
    -> Aggregate: sum((product_log.price_discount * product_log.order_count))  (cost=4.78 rows=17) (actual time=0.162..0.162 rows=1 loops=592913)
        -> Filter: ((product_log.create_at >= 1660590000) and (product_log.create_at <= 1663268400))  (cost=3.08 rows=17) (actual time=0.156..0.160 rows=30 loops=592913)
            -> Index lookup on product_log using ix-product_log-product_id-category_id (product_id=p.id)  (cost=3.08 rows=153) (actual time=0.131..0.153 rows=170 loops=592913)

if add order

-> Sort: t.revenue DESC  (actual time=200.365..234.906 rows=1123106 loops=1)
    -> Table scan on t  (cost=254074.08 rows=2258414) (actual time=0.001..25.554 rows=1123106 loops=1)
        -> Materialize  (cost=1246271.02..1246271.02 rows=2258414) (actual time=138502.280..138581.500 rows=1123106 loops=1)
            -> Nested loop inner join  (cost=1020429.62 rows=2258414) (actual time=0.021..1869.169 rows=1123106 loops=1)
                -> Index lookup on category_product using i_category_product (category_id=269)  (cost=229984.72 rows=2258414) (actual time=0.011..466.044 rows=1123106 loops=1)
                -> Single-row index lookup on p using PRIMARY (id=category_product.product_id)  (cost=0.25 rows=1) (actual time=0.001..0.001 rows=1 loops=1123106)
        -> Select #3 (subquery in projection; dependent)
            -> Aggregate: sum((product_log.price_discount * product_log.order_count))  (cost=4.76 rows=17) (actual time=0.121..0.121 rows=1 loops=1123106)
                -> Filter: ((product_log.create_at >= 1660590000) and (product_log.create_at <= 1663268400))  (cost=3.06 rows=17) (actual time=0.115..0.118 rows=30 loops=1123106)
                    -> Index lookup on product_log using ix-product_log-product_id-category_id (product_id=p.id)  (cost=3.06 rows=153) (actual time=0.090..0.112 rows=173 loops=1123106)

the problem, it seems to me, is mainly that sorting needs to be applied to the calculated field. mb description of the use of the request will give some additional information. The site has a page with product categories. some of the categories have a huge number of products, more than 2 million. going to the category, the user sees a table with fields and aggregated information on them for 1 month, in the future he will be able to change both the displayed period (week, month, 3 months) and the period frame, so it is simply unrealistic to calculate this data in advance for each option of these settings.

1 Answers

In general, avoid subqueries.

  • The outer nesting is unnecessary; simply tack the ORDER BY onto the second level.

  • IN ( SELECT ... ) is often poorly optimized. Some form of JOIN is usually better.

  • Do all the SUMs once, using a GROUP BY.

Something like

SELECT  p.id, s.revenue
    FROM (
        SELECT  pl.id, SUM(price_discount * order_count) AS revenue
            FROM  product_log AS pl
            GROUP BY  pl.id
            WHERE  pl.create_at >= 1660590000
              AND  pl.create_at <= 1663268400 
         ) AS s
    JOIN  category_product AS cp  ON cp.product_id = s.id
    WHERE  cp.category_id = 8712
    ORDER BY  s.revenue desc 

Plus indexes:

cp:  INDEX(category_id, product_id)
pl:  INDEX(create_at, id)

cp smells like a Many-to-many mapping table; follow the indexing advice there.

Perhaps you want "<" instead of "<=".

The use of a Summary Table would be the next step in improving performance.

Related