Elasticsearch remove special characters (from non ascii based language)

user_433570 0
Viewed 2030

For english, I could use something like

       "specialCharactersFilter": {
         "pattern": "[^A-Za-z0-9]",
         "type": "pattern_replace",
         "replacement": ""
       }
     }

to remove non-text characters.

However, for non-ascii language such as asian, the above filter removes all valid non-special characters.

How to remove special characters from asian language?

elasticsearch regex
1 Answers

It looks like the regex flavor used in pattern_replace filter is java.util.regex.

To remove any characters other than any Unicode letter and decimal digit, you may use

 "specialCharactersFilter": {
     "pattern": "[^\\p{L}\\p{Nd}]",
     "type": "pattern_replace",
     "replacement": ""
   }

To make sure you only keep ASCII digits and remove all Hindi, Tamil etc. digits, you may use a variation like

"pattern": "[^\\p{L}0-9]"

See the regex demo.

user_38329700
Related