Lucene ignores / overwrite fuzzy edit distance in QueryParser

user_204693 0
Viewed 46

Given the following QueryParser with a FuzzySearch term in the query string:

fun fuzzyquery() {
    val query = QueryParser("term", GermanAnalyzer()).parse("field:search~4")
    println(query)
}

The resulting Query will actually have this representation:

field:search~2

So, the ~4 gets rewritten to ~2. I traced the code down to the following implementation:

QueryParserBase

protected Query newFuzzyQuery(Term term, float minimumSimilarity, int prefixLength) {
    String text = term.text();
    int numEdits = FuzzyQuery.floatToEdits(minimumSimilarity, text.codePointCount(0, text.length()));
    return new FuzzyQuery(term, numEdits, prefixLength);
}

FuzzyQuery

public static int floatToEdits(float minimumSimilarity, int termLen) {
    if (minimumSimilarity >= 1.0F) {
        return (int)Math.min(minimumSimilarity, 2.0F);
    } else {
        return minimumSimilarity == 0.0F ? 0 : Math.min((int)((1.0D - (double)minimumSimilarity) * (double)termLen), 2);
    }
}

As is clearly visible, any value higher than 2 will always get reset to 2. Why is this and how can I correctly get the fuzzy edit distance I want into the query parser?

lucene
1 Answers

This may cross the border into "not an answer" - but it is too long for a comment (or a few comments):

Why is this?

That was a design decision, it would seem. It's mentioned in the documentation here.

"The value is between 0 and 2"

There is an old article here which gives an explanation:

"Larger differences are far more expensive to compute efficiently and are not processed by Lucene.".

I don't know how official that is, however.

More officially, from the JavaDoc for the FuzzyQuery class, it states:

"At most, this query will match terms up to 2 edits. Higher distances (especially with transpositions enabled), are generally not useful and will match a significant amount of the term dictionary."

How can I correctly get the fuzzy edit distance I want into the query parser?

You cannot, unless you customize the source code.

The best (least worst?) alternative, I think, is probably the one mentioned in the above referenced FuzzyQuery Javadoc:

"If you really want this, consider using an n-gram indexing technique (such as the SpellChecker in the suggest module) instead."

In this case, one price to be paid will be a potentially much larger index - and even then, n-grams are not really equivalent to edit distances. I don't know if this would meet your needs.

user_125673650
Related