Implementing custom prefix remover token filter in lucene producing dirty tokens

user_17394561 0
Viewed 65

i'm trying to implement a lucene filter to remove a prefix from a term in a query. It seems that sometime after multiple queries, the filter has been reused so the char buffer is dirty.

Code below is simplified, prefix is an external parameter.

  public static class PrefixFilter extends TokenFilter {

    private final PackedTokenAttributeImpl termAtt = (PackedTokenAttributeImpl) addAttribute(CharTermAttribute.class);

    public PrefixFilter(TokenStream in) {
      super(in);
    }

    @Override
    public final boolean incrementToken() throws IOException {
      if (!input.incrementToken()) {
        return false;
      }
      String value = new String(termAtt.buffer());
      value = value.trim();
      value = value.toLowerCase();
      value = StringUtils.removeStart(value, "prefix_");
      if (value.isBlank()) {
        termAtt.setEmpty();
      } else {
        termAtt.copyBuffer(value.toCharArray(), 0, value.length());
        termAtt.setLength(value.length());
      }
      return true;
    }
  }

So after 10 or twelve queries, the value "prefix_a" became "abcde".

So i'm trying to add termBuffer offset end value in this way:

    termAtt.setEmpty();
    termAtt.resizeBuffer(value.length());
    termAtt.copyBuffer(value.toCharArray(), 0, value.length());
    termAtt.setLength(value.length());
    termAtt.setOffset(0, value.length());

But i don't know if it's correct. Can anyone help me?

Thanks.

buffer lucene java
1 Answers

See if this helps you,

/**
 * Standard number token filter.
 */
public class StandardnumberTokenFilter extends TokenFilter {

    private final LinkedList<PackedTokenAttributeImpl> tokens;

    private final StandardnumberService service;

    private final Settings settings;

    private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);

    private final PositionIncrementAttribute posIncAtt = addAttribute(PositionIncrementAttribute.class);

    private State current;

    protected StandardnumberTokenFilter(TokenStream input, StandardnumberService service, Settings settings) {
        super(input);
        this.tokens = new LinkedList<>();
        this.service = service;
        this.settings = settings;
    }

    @Override
    public final boolean incrementToken() throws IOException {
        if (!tokens.isEmpty()) {
            if (current == null) {
                throw new IllegalArgumentException("current is null");
            }
            PackedTokenAttributeImpl token = tokens.removeFirst();
            restoreState(current);
            termAtt.setEmpty().append(token);
            posIncAtt.setPositionIncrement(0);
            return true;
        }
        if (input.incrementToken()) {
            detect();
            if (!tokens.isEmpty()) {
                current = captureState();
            }
            return true;
        } else {
            return false;
        }
    }

    private void detect() throws CharacterCodingException {
        CharSequence term = new String(termAtt.buffer(), 0, termAtt.length());
        Collection<CharSequence> variants = service.lookup(settings, term);
        for (CharSequence ch : variants) {
            if (ch != null) {
                PackedTokenAttributeImpl token = new PackedTokenAttributeImpl();
                token.append(ch);
                tokens.add(token);
            }
        }
    }

    @Override
    public void reset() throws IOException {
        super.reset();
        tokens.clear();
        current = null;
    }

    @Override
    public boolean equals(Object object) {
        return object instanceof StandardnumberTokenFilter &&
                service.equals(((StandardnumberTokenFilter)object).service) &&
                settings.equals(((StandardnumberTokenFilter)object).settings);
    }

    @Override
    public int hashCode() {
        return service.hashCode() ^ settings.hashCode();
    }
}

https://github.com/jprante/elasticsearch-plugin-bundle/blob/f63690f877cc7f50360faffbac827622c9d404ef/src/main/java/org/xbib/elasticsearch/plugin/bundle/index/analysis/standardnumber/StandardnumberTokenFilter.java

user_70646170
Related