ComboAnalyzer - AttributeImpl not found in AttributeSource

user_2219212 0
Viewed 29

The analyzer OpenNLPAnalyzer based on OpenNLPTokenizer in the opennlp package that ships with Lucene in this blog post works as promised. I am now trying to use it inside an ComboAnalyzer (a part of an ES-plugin to combine multiple analyzers; see link below) in the following way:

ComboAnalyzer analyzer = new ComboAnalyzer(new EnglishAnalyzer(), new OpenNLPAnalyzer());
TokenStream stream = analyzer.tokenStream("fieldname", new StringReader(text));

stream is a ComboTokenStream. On calling stream.incrementToken(), I get the following exception at line 105 here:

Exception in thread "main": State contains AttributeImpl of type org.apache.lucene.analysis.tokenattributes.OffsetAttributeImpl that is not in in this AttributeSource

Here is what the called method restoreState does.

public final void restoreState(State state) {
    if (state == null)  return;

    do {
        AttributeImpl targetImpl = attributeImpls.get(state.attribute.getClass());
        if (targetImpl == null) {
            throw new IllegalArgumentException("State contains AttributeImpl of type " +
                    state.attribute.getClass().getName() + " that is not in in this AttributeSource");
        }
        state.attribute.copyTo(targetImpl);
        state = state.next;
    } while (state != null);
}

This hints that one of the TokenStreams has an OffsetAttribute but the other does not. Is there a clean way to fix this?

I tried to add the line addAttribute(OffsetAttribute.class) in the same file here. I still get the same exception.

lucene java
1 Answers

The problem was here:

Tokenizer source = new OpenNLPTokenizer(
                    AttributeFactory.DEFAULT_ATTRIBUTE_FACTORY, sentenceDetectorOp, tokenizerOp);

The fix is to pass in TokenStream.DEFAULT_TOKEN_ATTRIBUTE_FACTORY instead of AttributeFactory.DEFAULT_ATTRIBUTE_FACTORY. The former uses PackedTokenAttributeImpl for implementing OffsetAttribute (and many other attributes) and the latter picks OffsetAttributeImpl.

user_22192120
Related