How to get the word boundary in ZZ_CMAP_PACKED?

Question

I am trying to write my first Elasticsearch Analysis Plugin, and I found a Extensiable Standard Analyser Plugin project from github:elasticsearch-analysis-standardext, in the project it provides codes below:

   /** 
    * Word Boundary "character classes" that are used in the ZZ_CMAP_PACKED 
    * string 
    */ 
   public static final Character WB_CLASS_BREAK               = '\0'; 
   public static final Character WB_CLASS_LETTER              = '\176'; 
   public static final Character WB_CLASS_NUMERIC             = '\200'; 
   public static final Character WB_CLASS_MID_LETTER          = '\202'; 
   public static final Character WB_CLASS_MID_NUMBER          = '\203'; 
   public static final Character WB_CLASS_MID_NUMBER_LETTER   = '\204'; 
   public static final Character WB_CLASS_EXTENDED_NUM_LETTER = '\205'; 
   public static final Character WB_CLASS_SINGLE_QUOTE        = '\212'; 
   public static final Character WB_CLASS_DOUBLE_QUOTE        = '\213';

Here is the link

I want to know how to get these word boundaries' character classes.

I ask the author with issue: How to get word boundaries "character classes"? #2, but it seems that the author won't answer my question.

I try to read the Unicode Text Segmentation documents: https://www.unicode.org/reports/tr29/ , but I really can't found what I want since it is hard and long to read.

I download the Standard Analyser Code from lucene repository, it is here: https://github.com/apache/lucene-solr/tree/releases/lucene-solr/8.1.0/lucene/core/src/java/org/apache/lucene/analysis/standard

It seems that the StandardTokenizerImpl.java class is generated from StandardTokenizerImpl.jflex, can I get these word boundaries' character classes with the help of jfex

Thanks for someone helps me.

user_11088913 · Answer

I found that we can get this world boundaries from the document: https://www.unicode.org/reports/tr29/#Word_Boundaries, and since the project elasticsearch-analysis-standardext is really old version of elasticsearch. It won't work with Elasticsearch version = 7.3.2.

And we can found that now Lucene don't support simple world boundaries like ALetter, it's format will be

MidLetterEx         = [\p{WB:MidLetter}\p{WB:MidNumLet}\p{WB:SingleQuote}]

we can see that from the file StandardTokenizerImpl.jflex

So if you want to get the world boundary from the ZZ_CMAP_PACKED, you should refer to the document Word_Boundaries.

May this will help you if you want to write an custom StandardTokenizer.

Also you can refer to the ClassicTokenizer from Lucene project, it will help you!

How to get the word boundary in ZZ_CMAP_PACKED?

1 Answers