Why Elasticsearch Arabic analyzer comes alongside Persian analyzer in the Persian examples?

user_9099894 0
Viewed 520

I recently checked the language analyzer in elasticsearch docs and wonder why in this example: analysis-lang-analyzer the Persian analyzer comes after the Arabic analyzer?! Is that needed because of anything? I mean the Persian analyzer alone, is not enough for the Persian language?

Persian analyzer The Persian analyzer could be reimplemented as a custom analyzer as follows:

PUT /persian_example
{
  "settings": {
    "analysis": {
      "char_filter": {
        "zero_width_spaces": {
            "type":       "mapping",
            "mappings": [ "\\u200C=>\\u0020"] 
        }
      },
      "filter": {
        "persian_stop": {
          "type":       "stop",
          "stopwords":  "_persian_" 
        }
      },
      "analyzer": {
        "rebuilt_persian": {
          "tokenizer":     "standard",
          "char_filter": [ "zero_width_spaces" ],
          "filter": [
            "lowercase",
            "decimal_digit",
            "arabic_normalization",
            "persian_normalization",
            "persian_stop"
          ]
        }
      }
    }
  }
}
elasticsearch
2 Answers

I don't think its required, you can index few sentences by removing the arabic_normalization and use the analyze API to check the tokens generated by persian analyzer and see if it generates the correct expected tokens.

You can also open the issue on elastic repo https://github.com/elastic/elasticsearch/issues which is the best place to ask this question as there someone from elastic can comment whether its a documentation issue or some real issue.

user_40394310

this analyzer are made by Lucene Users and forum. you should check what arabic_normalization do in lucene and is it necessary?

this is arabic_normalization class description:

Normalizer for Arabic.

Normalization is defined as:

Normalization of hamza with alef seat to a bare alef.

Normalization of teh marbuta to heh

Normalization of dotless yeh (alef maksura) to yeh.

Removal of Arabic diacritics (the harakat) Removal of tatweel (stretching character).

(as I know Persian) I think it is better for Persian indexing that you first use Arabic Normalizer

user_97921040
Related