A better approach to exclude large list of items in Elasticsearch

user_5470728 0
Viewed 921

I use terms query to exclude a list of 100,000 or more items, as the terms query by default allows only 65,536 terms, ES throws following error:

The number of terms [115687] used in the Terms Query request has exceeded the allowed maximum of [65536]. This maximum can be set by changing the [index.max_terms_count] index level setting.

One way to solve my problem is to increase the max_terms_count, but I suspect it will be slow.

Another solution would be to exclude those items in PHP which also would be too resource consuming.

Is there a better way to exclude large list of items from ES search result?

elasticsearch php
2 Answers
  1. For rare cases I suggest to use a client-oriented solution: split exceptions list into two: the first 65k items should be processed by ES, the rest - in PHP.
  2. Performance-oriented solution: limit the exclusion list to 65k (client-side limitation)
user_117340020

I found a solution, I hope it will help to those who might face the same kind of issue. The solution is to split the large list into multiple smaller lists and put each of them in separate "terms" queries. For example, let's say that the max_terms_count is 4 and we have 12 items which we need to exclude from the search result.

$iMaxTermsCount = 4;

$arItems = [
    1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
];

if (count($arItems) <= $iMaxTermsCount) {
    // Add all items to the terms query. It will produce:
    // "terms": {
    //  "item_id": [
    //      1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
    //  ]
    // }
} else {
    $arChunks = array_chunk($arItems, $iMaxTermsCount);

    foreach ($arChunks as $ar) {
        // Add some (4) items to the terms query.
    }

    // The loop above will produce:
    // {
    //  "terms": {
    //      "item_id": [
    //          1, 2, 3, 4
    //      ]
    //  }
    // },
    // {
    //  "terms": {
    //      "item_id": [
    //          5, 6, 7, 8
    //      ]
    //  }
    // },
    // {
    //  "terms": {
    //      "item_id": [
    //          9, 10, 11, 12
    //      ]
    //  }
    // }
}

The final JSON object will be like this:

{
    "query": {
        "bool": {
            "must_not": [
                {
                    "bool": {
                        "should": [
                            {
                                "terms": {
                                    "item_id": [
                                        1, 2, 3, 4
                                    ]
                                }
                            },
                            {
                                "terms": {
                                    "item_id": [
                                        5, 6, 7, 8
                                    ]
                                }
                            },
                            {
                                "terms": {
                                    "item_id": [
                                        9, 10, 11, 12
                                    ]
                                }
                            }
                        ]
                    }
                }
            ]
        }
    }
}

The query above will exclude all items without throwing any errors.

user_54707280
Related