How to omit similar results in elastic search DSL query just like other search engines?

user_5275677 0
Viewed 423

I'm using ElasticSearch for storing some documents to create a search engine, everything works perfectly.

we may store similar documents in our elastic cloud.

I can see that some search engines omit similar data like Google.

Check out this message at the bottom of Google's result:

enter image description here

I need to implement exactly this, I want to know is there any way to omit similar documents in query results using ElasticSearch and give this option to the user that he either wants to see all results or not?

any suggestions, please?

UPDATE

I decided to add more information to make it clear. consider these two documents below:

Node.js is a JavaScript runtime built on Chrome's V8 JavaScript engine.

Node.js is a platform built on Chrome's V8 JavaScript engine.

there are a lot of documents like these in my elastic index. how can I omit similars and just show one of them in the DSL query?

is there any way to achieve this or need AI, ML, and these kinds of stuff to implement this?

elastic-stack elasticsearch
2 Answers

You can use collapse parameter to collapse search results based on field values for example, the following search collapses results by user.id and sorts them by http.response.bytes.

GET /my-index-000001/_search
{
  "query": {
    "match": {
      "message": "GET /search"
    }
  },
  "collapse": {
    "field": "user.id"                
  },
  "sort": [ "http.response.bytes" ],  
  "from": 10                          
}

For more information you can check Elastic official documentation here.

user_140183850

In order to solve your issue, you need to define what do you mean by "similar".

Google has a custom definition for it, that works for webpages. That definition is based on domains - you know, not to get flooded by results from one domain, but also based on the content - Google ranks low copied webpages.

You need to define a metrics that is specific for your documents. It is usually based on one specific property, the determinant field. What will be your determinant- you know it. If you have a forum engine, the topicID is a good choice. If you have a product catalog, the leaf categoryID will be the best.

There are two ways to achieve this goal.

  1. Collapsing

This does exactly what you want. You can define the determinant field in your document, you can define what you want to show as a detail, etc. The ElasticSearch documentation gives you lots of hints.

This will give you exact results, no matter how many shards you have, though it's a bit heavier operation.

  1. Aggregation + Sampling

In this solution you create an aggregation, and use a sampler to pick some sample documents for you. Something like this:

    "aggregations": {
        "sample": {
            "diversified_sampler": {
                "field": "topicId",
                "shard_size": 200,
                "max_docs_per_value": 1
            },

This solution is faster, because - unlike collapse - it does not issue subqueries for all groups. The disadvantage is that this solution may not count exactly when using multiple shards. Also, you cannot specify ordering for the sampler - it always works by relevance.

There is an interesting discussion about comparing the two approaches on an ElasticSearch forum, worth to read it through.

user_18912190
Related