MySQL Subquery making query super slow

Question

I've been struggling when it comes to optimizing the following query (Example 1):

    SELECT `service`.* 
    FROM
    (
          SELECT `storeUser`.`storeId`
          FROM `storeUser`
          WHERE `storeUser`.`userId` = 1
      UNION
          SELECT `store`.`storeId`
          FROM `companyUser`
            INNER JOIN `store` ON `companyUser`.`companyId` = `store`.`companyId`
          WHERE `companyUser`.`userId` = 1
      UNION
      SELECT `store`.`storeId`
      FROM `accountUser`
        INNER JOIN `company` ON `company`.`accountId` = `accountUser`.`accountId`
        INNER JOIN `store` ON `company`.`companyId` = `store`.`companyId`
      WHERE `accountUser`.`userId` = 1
    ) AS `storeUser`

        INNER JOIN `service` ON `storeUser`.`storeId` = `service`.`storeId`
    LIMIT 10;

The subquery should be returning something like "1","2","3,"4"

Anyway it's super slow and takes about 48 seconds to give a response, even though the subquery by itself, ran in a different console, takes about 0,0020ms to give results.

The same applies if I place the subquery inside an IN instead (Example 2):

    SELECT `service`.*
    FROM `service`
    WHERE 1
    AND `service`.`storeId` IN (
        SELECT `storeUser`.`storeId` FROM `storeUser` WHERE `storeUser`.`userId` = 1
      UNION
        SELECT `store`.`storeId` FROM `companyUser` 
            INNER JOIN `store` ON `companyUser`.`companyId` = `store`.`companyId`
        WHERE `companyUser`.`userId` = 1
      UNION
        SELECT `store`.`storeId`
        FROM `accountUser`
            INNER JOIN `company` ON `company`.`accountId` = `accountUser`.`accountId`
            INNER JOIN `store` ON `company`.`companyId` = `store`.`companyId`
      WHERE `accountUser`.`userId` = 1
    )
    LIMIT 10;

However if I simply put the values returned by that query, manually, it's basically instantly:

    SELECT
      `service`.*
    FROM
      `service`
    WHERE 1
    AND `service`.`storeId` IN (
      "1", "2", "3", "4", "5"
    )
    LIMIT 10;

Important to mention that'd I've reviewed the indexes in the joins and everything seems to be in place, and the EXPLAIN [query] returns a filtered score of 100 for basically everything.

Edit:

Sorry for not providing enough information before, hope this can be more helpful:

MySQL 5.7,
Storage engine: InnoDB

EXPLAINs
1.) StoreUser

id | select_type | table          | partitions | type | possible_keys          |  key     | key_len | ref   | rows  | filtered | Extra
1  | SIMPLE      | storeUser      | NULL       | ref  | PRIMARY, storeUserUser |  PRIMARY | 4       | const | 1     |100.00    | Using index

2.) CompanyUser

id  | select_type | table       | partitions | type | possible_keys                              | key              | key_len | ref                         | rows  | filtered  | Extra
1   | SIMPLE      | companyUser | NULL       | ref  | PRIMARY,companyUserCompany,companyUserUser | companyUserUser  | 4       | const                       | 30    | 100.00    | Using index
1   | SIMPLE      | store       | NULL       | ref  | storeCompany                               | storeCompany     | 4       | Table.companyUser.companyId | 5     | 100.00    | Using index

3.) AccountUser

id  | select_type | table         | partitions | type | possible_keys           | key             | key_len | ref                         | rows  | filtered  | Extra
1   | SIMPLE      | accountUser   | NULL       | ref  | PRIMARY,accountUserUser | accountUserUser | 4       | const                       | 1     | 100.00    | Using index
1   | SIMPLE      | company       | NULL       | ref  | PRIMARY,companyAccount  | companyAccount  | 4       | Table.accountUser.accountId | 305   | 100.00    | Using index
1   | SIMPLE      | store         | NULL       | ref  | storeCompany            | storeCompany    | 4       | Table.company.companyId     | 5     | 100.00    | Using index

4.) Whole query (Example 2)

id    | select_type         | table       | partitions  | type    | possible_keys                               | key             | key_len | ref                         | rows    | filtered  | Extra
1     | PRIMARY             | service     | NULL        | ALL     | NULL                                        | NULL            | NULL    | NULL                        | 2836046 | 100.00    | Using where
2     | DEPENDENT SUBQUERY  | storeUser   | NULL        | eq_ref  | PRIMARY,storeUserStore,storeUserUser        | PRIMARY         | 8       | const,func                  | 1       | 100.00    | Using index
3     | DEPENDENT UNION     | store       | NULL        | eq_ref  | PRIMARY,storeCompany                        | PRIMARY         | 4       | func                        | 1       | 100.00    | NULL
3     | DEPENDENT UNION     | companyUser | NULL        | eq_ref  | PRIMARY,companyUserCompany,companyUserUser  | PRIMARY         | 8       | const,Table.store.companyId | 1       | 100.00    | Using index
4     | DEPENDENT UNION     | companyUser | NULL        | ref     | PRIMARY,accountUserUser                     | accountUserUser | 4       | const                       | 1       | 100.00    | Using index
4     | DEPENDENT UNION     | store       | NULL        | eq_ref  | PRIMARY,storeCompany                        | PRIMARY         | 4       | func                        | 1       | 100.00    | NULL
4     | DEPENDENT UNION     | company     | NULL        | eq_ref  | PRIMARY,companyAccount                      | PRIMARY         | 4       | Table.store.companyId       | 1       | 100.00    | Using where
NULL  | UNION RESULT        | | NULL        | ALL     | NULL                                        | NULL            | NULL    | NULL                        | NULL    | NULL      | Using temporary

user_205608 · Answer

You didn't show us your indexes or EXPLAIN output, so all this is guesswork.

Clearly it's the subquery in your second example that's not optimized. That subquery is a UNION with three branches. The way you address performance trouble? Analyze and optimize each branch of the UNION separately.

You certainly need some better indexes, unless your database server is too small or misconfigured. That's very rare, so let's work on indexes.

The first branch is

 SELECT storeUser.storeId
   FROM storeUser
  WHERE storeUser.userId = 1

This compound index covers that query. Try adding it. If you have a separate index on just userId, drop it when you add this one.

ALTER TABLE storeUser ADD INDEX userId_storeId (userId, storeId);

The second branch is

 SELECT store.storeId
   FROM companyUser
  INNER JOIN store  ON companyUser.companyId = store.companyId
  WHERE companyUser.userId = 1

Subqueries with JOIN operations are a little tricker to optimize without access to EXPLAIN output, so this is guesswork. I guess these indexes will help, though. (Assuming you use InnoDB and the PK on store is storeId.)

ALTER TABLE companyUser ADD INDEX userId_companyId (userId, companyId);
ALTER TABLE store ADD INDEX companyId (companyId);

Similar analysis applies to the third branch of the UNION.

And, add this index. Your EXPLAIN points to it being missing, and so a full table scan of that large table being required.

ALTER TABLE service ADD INDEX storeId (storeId);

Again, helping you would be far easier if you showed us your table definitions with indexes. SHOW CREATE TABLE service; for example, would show us what we need for your service table. Pro tip when troubleshooting this kind of performance stuff always doublecheck your indexes. Ask me how I know that when you have a couple of hours to spare.

Pro tip Be obsessive about formatting your queries so they're readable. You, yourself a year from now, and your co-workers yet unborn need to read and reason about them. To my way of thinking that means skipping those silly backticks.

user_1766831 · Answer

Perhaps you need to rethink the schema. It seems like you need a table for "user" instead of, or in addition to, the 3 tables for different types of "users".

Meanwhile, these composite indexes are likely to help performance in either formulation:

storeUser:  INDEX(storeId,  userId)
storeUser:  INDEX(userId,  storeId)
service:  INDEX(storeId)
store:  INDEX(companyId,  storeId)
companyUser:  INDEX(userId, companyId)
company:  INDEX(accountId,  companyId)
accountUser:  INDEX(userId, accounted)

When adding a composite index, DROP index(es) with the same leading columns. That is, when you have both INDEX(a) and INDEX(a,b), toss the former.

In particular, storeUser smells like a many-to-many mapping table. If so, see Many:many mapping for more discussion.

In general IN( SELECT ... ) does not optimize well, but you might find otherwise for your query.

user_7890266 · Answer

Sorry to not give more details about the schemas but I wasn't allowed to share it here, anyway, the problem happened to be elsewhere: The service table was receiving a huge amount of requests, some actions were even locking it up, ending up on slow times whenever we were accesing that table, we have fixed our other proccess and it's working great now. Hugely appreciate your time and effort, thanks.

MySQL Subquery making query super slow

3 Answers