elk's elastic search dsl case sensitive

user_365251 0
Viewed 290

I'm doing an Elasticsearch Query DSL query on ELK such as:

{
  "query": {
    "wildcard": {
      "url.path": {
        "value": "*download*",
        "boost": 1,
        "rewrite": "constant_score"
      }
    }
  }
}

but it seems is case sensitive (so show only info with "download", not "Download" or "DOWNLOAD"). i.e. is case sensitive.

can I disable this? and search case insensitive?

Version used: 7.9.1

elk elasticsearch dsl case-sensitive
3 Answers

The below query will help you perform case-insensitive search as it will fetch results for *download, *Download and *DOWNLOAD. You may replace with your index and with the field you would like to perform this search.

Search Query

   GET /<my-index>/_search
    {
     "query" : {
         "bool" : {
            "must" : {
               "query_string" : {
                   "query" : "*download",
                   "fields": ["<field1>"]
               }
            }
         }
     }
}

If you wish to perform the same search on multiple fields, you can add the same in list.

Search on multiple fields

GET /<my-index>/_search
    {
     "query" : {
         "bool" : {
            "must" : {
               "query_string" : {
                   "query" : "*download",
                   "fields": ["<field1>","<field2>","field3>"]
               }
            }
         }
     }
}
user_88507020

There is a case_insensitive parameter available for wildcard query, but it was introduced in Elasticsearch 7.10.0, so you need to upgrade if you are still on 7.9.1.

If you can upgrade to 7.10.0 or higher:

Ideally, in index mapping field should use wildcard type:

{
  "mappings": {
    "properties": {
      "url.path": {
        "type": "wildcard"
      }
    }
  }
}

Then a wildcard query with case insensitivity enabled will find all the variants ("download", "DOWNLOAD", "download", etc)

{
  "query": {
    "wildcard": {
      "url.path": {
        "value": "*download*",
        "boost": 1,
        "rewrite": "constant_score",
        "case_insensitive": true
      }
    }
  }
}

If you must remain at 7.9.1:

Define your mapping in such a way that Elasticsearch treats the field contents as lowercase. The following will mimic wildcard type (it's a keyword, so only one token) indexed as lowercase.

{
  "mappings": {
    "properties": {
      "url": {
        "type": "text",
        "analyzer": "lowercase-keyword"
      }
    }
  },
  "settings": {
    "analysis": {
      "analyzer": {
        "lowercase-keyword": {
          "type": "custom",
          "tokenizer": "keyword",
          "filter": "lowercase"
        }
      }
    }
  }
}

The query, without the case_insensitive parameter which is unsupported in this version:

{
  "query": {
    "wildcard": {
      "url": {
        "value": "*download*",
        "boost": 1,
        "rewrite": "constant_score"
      }
    }
  }
}

Example results (note that searching for "*download*" and "*DoWnLoAd*" with both work in the same way):

{
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "my-index",
        "_type": "_doc",
        "_id": "PtbQe3wByTvslqtrs7Cn",
        "_score": 1.0,
        "_source": {
          "url": "http://example.com/download"
        }
      },
      {
        "_index": "my-index",
        "_type": "_doc",
        "_id": "P9bQe3wByTvslqtrvbDt",
        "_score": 1.0,
        "_source": {
          "url": "http://example.com/Download"
        }
      },
      {
        "_index": "my-index",
        "_type": "_doc",
        "_id": "QNbQe3wByTvslqtrzbDw",
        "_score": 1.0,
        "_source": {
          "url": "http://example.com/DOWNLOAD"
        }
      }
    ]
  }
}
user_3698940

You can use case_insensitive parameter for wildcard query. This parameter was introduced in 7.10.0 version

Adding a working example with index data, mapping, search query, and search result

Index Mapping:

{
  "mappings": {
    "properties": {
      "url": {
        "properties": {
          "path": {
            "type": "wildcard"
          }
        }
      }
    }
  }
}

Index Data:

{
  "url":{
    "path":"xx/download"
  }
}

Search Query:

{
  "query": {
    "wildcard": {
      "url.path": {
        "value": "*Download*",
        "boost": 1,
        "rewrite": "constant_score",
         "case_insensitive": false
      }
    }
  }
}

Search Result:

No results will be there when you are searching for *Download* or *DOWNLOAD*

Update:

You can use the wildcard query with "case_insensitive": true parameter

Adding a sample index data, search query, and search result

Index Data:

{
  "url": {
    "path": "download"
  }
}
{
  "url": {
    "path": "DOWNLOAD"
  }
}
{
  "url": {
    "path": "Download"
  }
}

Search Query:

{
  "query": {
    "wildcard": {
      "url.path": {
        "value": "*DOWNLOAD*",
        "boost": 1,
        "rewrite": "constant_score",
        "case_insensitive": true
      }
    }
  }
}

Search Result:

"hits": [
      {
        "_index": "67210888",
        "_type": "_doc",
        "_id": "1",
        "_score": 1.0,
        "_source": {
          "url": {
            "path": "download"
          }
        }
      },
      {
        "_index": "67210888",
        "_type": "_doc",
        "_id": "2",
        "_score": 1.0,
        "_source": {
          "url": {
            "path": "Download"
          }
        }
      },
      {
        "_index": "67210888",
        "_type": "_doc",
        "_id": "3",
        "_score": 1.0,
        "_source": {
          "url": {
            "path": "DOWNLOAD"
          }
        }
      }
    ]
user_103487580
Related