MongoDB aggregate index not being used without hint

Viewed 761

I have a collection of fruit carts, millions of rows (names changed to protect the guilty.)

Each of these carts has an owner, a make and a model, as well as several Y or N fields on whether the cart contains certain types of fruits.

I would like to generate a list of models and a count of how many carts there are of each that have certain types of fruit.

So the data would be like:

fruit.cart.insert(
{ 'owner' => 'Fred',   'make' => 'Toshiba', 'model' => 'fruitmaster 9000', 'have_apples' => 'Y', 'have_grapes' => 'N', 'have_peaches' => 'Y', ... },
{ 'owner' => 'Wilma',  'make' => 'Toshiba', 'model' => 'fruitmaster 9000', 'have_apples' => 'Y', 'have_grapes' => 'N', 'have_peaches' => 'N', ... },
{ 'owner' => 'Betty',  'make' => 'Toshiba', 'model' => 'fruitmaster 9000', 'have_apples' => 'N', 'have_grapes' => 'Y', 'have_peaches' => 'Y', ... },
{ 'owner' => 'Barney', 'make' => 'Toshiba', 'model' => 'fruitmaster X5',   'have_apples' => 'N', 'have_grapes' => 'N', 'have_peaches' => 'Y', ... },
{ 'owner' => 'Fred',   'make' => 'Honda',   'model' => 'T-1000',           'have_apples' => 'Y', 'have_grapes' => 'Y', 'have_peaches' => 'Y', ... },
{ 'owner' => 'Wilma',  'make' => 'Honda',   'model' => 'T-1000',           'have_apples' => 'N', 'have_grapes' => 'N', 'have_peaches' => 'N', ... },

And I want the output to be:

{ 'make' => 'Toshiba', 'model' => 'fruitmaster 9000', 'count' => 3, 'apples_count' => 2, 'grapes_count' => 1, 'peaches_count' => 2, ... },
{ 'make' => 'Toshiba', 'model' => 'fruitmaster X5',   'count' => 1, 'apples_count' => 0, 'grapes_count' => 0, 'peaches_count' => 1, ... },
{ 'make' => 'Honda',   'model' => 'T-1000',           'count' => 2, 'apples_count' => 1, 'grapes_count' => 1, 'peaches_count' => 1, ... },

So here's my aggregate query:

{ '$group' =>
  {
    _id             =>  { 'model'  => '$model', },
    model           =>  { '$first' => '$model' },
    make            =>  { '$first' => '$make'  },
    count           =>  { '$sum'   =>  1 },
    oranges_count   =>  { '$sum'   => { '$cond' => [ { '$eq' => [ '$have_oranges', 'Y' ] }, 1, 0 ] }, },
    grapes_count    =>  { '$sum'   => { '$cond' => [ { '$eq' => [ '$have_grapes',  'Y' ] }, 1, 0 ] }, },
    peaches_count   =>  { '$sum'   => { '$cond' => [ { '$eq' => [ '$have_peaches', 'Y' ] }, 1, 0 ] }, },
    apples_count    =>  { '$sum'   => { '$cond' => [ { '$eq' => [ '$have_apples',  'Y' ] }, 1, 0 ] }, },
    pears_count     =>  { '$sum'   => { '$cond' => [ { '$eq' => [ '$have_pears',   'Y' ] }, 1, 0 ] }, },
  },
},

With index:

db.cart.createIndex (
 {
   'have_pears'    : 1,
   'have_apples'   : 1,
   'have_oranges'  : 1,
   'have_grapes'   : 1,
   'have_peaches'  : 1,
   'make'          : 1,
   'model'         : 1,
 }, { background : true, name : 'fruit_counts' } );

If I run this without providing the index hint then it doesn't use it and the query takes forever:

  ...
  'queryPlanner' => {
    'indexFilterSet' => bless( do{\(my $o = 0)}, 'boolean' ),
    'namespace' => 'fruit.cart',
    'parsedQuery' => {},
    'plannerVersion' => 1,
    'rejectedPlans' => [],
    'winningPlan' => {
      'direction' => 'forward',
      'stage' => 'COLLSCAN'
    }
  }
  ...

With hint it is fast:

  {
    'ok' => '1',
    'stages' => [
      {
        '$cursor' => {
          'fields' => {
            '_id' => 0,
            'have_peaches' => 1,
            'have_apples' => 1,
            'have_oranges' => 1,
            'have_grapes' => 1,
            'have_pears' => 1,
            'model' => 1,
            'make' => 1
          },
          'query' => {},
          'queryPlanner' => {
            'indexFilterSet' => bless( do{\(my $o = 0)}, 'boolean' ),
            'namespace' => 'fruit.cart',
            'parsedQuery' => {},
            'plannerVersion' => 1,
            'rejectedPlans' => [],
            'winningPlan' => {
              'inputStage' => {
                'direction' => 'forward',
                'indexBounds' => {
                  'have_peaches' => [ '[MinKey, MaxKey]' ],
                  'have_apples' => [ '[MinKey, MaxKey]' ],
                  'have_oranges' => [ '[MinKey, MaxKey]' ],
                  'have_grapes' => [ '[MinKey, MaxKey]' ],
                  'have_pears' => [ '[MinKey, MaxKey]' ],
                  'model' => [ '[MinKey, MaxKey]' ],
                  'make' => [ '[MinKey, MaxKey]' ]
                },
                'indexName' => 'fruit_counts',
                'indexVersion' => 2,
                'isMultiKey' => $VAR1->[0]{'stages'}[0]{'$cursor'}{'queryPlanner'}{'indexFilterSet'},
                'isPartial' => $VAR1->[0]{'stages'}[0]{'$cursor'}{'queryPlanner'}{'indexFilterSet'},
                'isSparse' => $VAR1->[0]{'stages'}[0]{'$cursor'}{'queryPlanner'}{'indexFilterSet'},
                'isUnique' => $VAR1->[0]{'stages'}[0]{'$cursor'}{'queryPlanner'}{'indexFilterSet'},
                'keyPattern' => {
                  'have_peaches' => '1',
                  'have_apples' => '1',
                  'have_oranges' => '1',
                  'have_grapes' => '1',
                  'have_pears' => '1',
                  'model' => '1',
                  'make' => '1'
                },
                'multiKeyPaths' => {
                  'have_peaches' => [],
                  'have_apples' => [],
                  'have_oranges' => [],
                  'have_grapes' => [],
                  'have_pears' => [],
                  'model' => [],
                  'make' => []
                },
                'stage' => 'IXSCAN'
              },
              'stage' => 'PROJECTION',
              'transformBy' => {
                '_id' => 0,
                'have_peaches' => 1,
                'have_apples' => 1,
                'have_oranges' => 1,
                'have_grapes' => 1,
                'have_pears' => 1,
                'model' => 1,
                'make' => 1
              }
            }
          }
        }
      },

...

documentdb without hint:

'queryPlanner' => {
  'namespace' => 'fruit.cart',
  'plannerVersion' => 1,
  'winningPlan' => {
    'inputStage' => {
      'inputStage' => {
        'stage' => 'COLLSCAN'
      },
      'stage' => 'SORT'
    },
    'stage' => 'SORT_AGGREGATE'
  }
},

documentdb with hint:

MongoDB::DatabaseError: Cannot use Hint for this Query. Index is multi key index or sparse index and query is not optimized to use this index.

DOH!!!

So what can I do to make it so the hint is not even required, and then make it work in DocumentDB?

1 Answers

The error is misleading. DocumentDB can use multi-key index hints, but the query itself must contain the fields that match the index, or you will see this message. You will see the same message even if you hint at a single key index, but the query does not contain the indexed field.

Related