How to sort with case insensitive without changing the settings

Admin 1
Viewed 690

My index name is data_new

Below is the code to insert into index

test = [   {'id':1,'name': 'A', 'professor': ['Bill Cage', 'accounting']},
    { 'id':2,  'name': 'AB', 'professor': ['Gregg Payne', 'engineering']},
    {'id':3,   'name': 'a',   'professor': ['Bill Cage', 'accounting']},
    {'id':4,'name': 'Tax Accounting 200', 'professor': ['Thomas Baszo', 'finance']},
    {'id':5,'name': 'Capital Markets 350', 'professor': ['Thomas Baszo', 'finance']},
    {'id':6,'name': 'Theatre 410', 'professor': ['Sebastian Hern', 'art']},
    {'id':7,'name': 'Accounting 101', 'professor': ['Thomas Baszo', 'finance']},
    {'id':8,'name': 'Marketing 101', 'professor': ['William Smith', 'finance']},
    {'id':8,'name': 'Anthropology 230', 'professor': ['Devin Cranford', 'history']},
    {'id':10,   'name': 'Computer Science 101',
        'professor': ['Gregg Payne', 'engineering']}]
from elasticsearch import Elasticsearch
import json
es = Elasticsearch()
es.indices.create(index='data_new', ignore=400)
for e in test:
        es.index(index="data_new", body=e, id=e['id'])
search = es.search(index="data_new", body={"from" : 0, "size" : 2,"query": {"match_all": {}}})
search['hits']['hits']

Right now

[{'id':1,'name': 'A'},
{ 'id':2,  'name': 'AB'},
{'id':3,   'name': 'a'}]

Expected is in below order

[{'id':1,'name': 'A'},
{ 'id':3,  'name': 'a'},
{'id':2,   'name': 'AB'}]

for input ["a", "b", "B", "C", "c", "A"] the result is : ["A", "B", "C", "a", "b", "c"]

I want output as ["A", "a", "B", "b", "C", "c"]

Expected out

My first Expected output > I need to sort the output with respect to name only in {Case insensitive}. I need to normalise name keyword and sort

How to do the modification on search = es.search(index="data_new", body={"from" : 0, "size" : 2,"query": {"match_all": {}}})

I have updated the code with below search = es.search(index="data_new", body={ "sort" : [{"name.keyword" : {"order" : "asc"}], {"size": 1000, "query": {"query_string": {"query": "A"}}})

with "normalizer": "case_insensitive"} I got the error

RequestError: RequestError(400, 'x_content_parse_exception', '[1:41] [field_sort] unknown field [normalizer]')

elasticsearch
2 Answers

In order to use normalizer, you need to define it into your mapping. you are not able to use it as an argument in your search. In your case, you need to have two fields for sort. I have made this by copying data to other fields. the first field has lowercase normalizer and the other one not.

PUT /test_index/
{
  "settings": {
    "analysis": {
      "normalizer": {
        "myLowercase": {
          "type": "custom",
          "filter": [ "lowercase" ]
        }
      }
    }
  },
  "mappings":{
     "post":{
        "properties":{
           "name":{
              "normalizer":"myLowercase",
              "type":"keyword",
              "copy_to": [
              "name2"
            ]
           },
           "name2":{
              "type":"keyword"
           }
        }
     }
  }
}

And your query would be something like this:

GET test_index/_search
{
  "query": {
    "match_all": {}
  },"sort": [
    {
      "name": {
        "order": "asc"
      }
    },
    {
      "name2":{
        "order": "asc"
      }
    }
  ]
}

This is the mapping and setting that you must have for your name field in your indices and you need to add other fields to the mapping as well. Please have the attention that this is for elasticsearch version below 7. If you use elasticsearch version 7 you must delete doc_type which is named post here from the mapping.

user_50889990

In order to do this you will have to use a script with ctx._source.mykey.toLowerCase() https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-sort-context.html

You can find another post which talk about it: Script-based sorting on Elasticsearch date field And a good article with an example here: https://qbox.io/blog/how-to-painless-scripting-in-elasticsearch

Code will look like (not tested)

{
 "query": {
   "match_all": {}
 },
 "sort": {
   "_script": {
     "type": "string",
     "order": "asc",
     "script": {
       "lang": "painless",
       "inline": "doc['name.keyword'].value.toLowerCase()"
     }
   }
 }
}

Note: It's a bad practice and you should do it only for a one shot query. If you want your application to stay healthy you should implement the solution suggested by saeednasehi. You can also use index sorting to be more performant.

user_25639460
Related