Apache solr fuzzy search with distance parameter as 2

Viewed 267

Enviornment- java version "11.0.12" 2021-07-20 LTS, solr-8.9.0

I have loaded a csv file in solr. csv file has a field 'name'. Type of 'Name' column in solr is defined as 'text_general'.

I understand that to perform a fuzzy search, tilde ~ symbol at the end of a single-word term is used. Default value of distance parameter is 2.

I have used following fuzzy-search query

http://localhost:8983/solr/startsolr/select?indent=on&wt=json&q=(Name:'Ellyse~') AND (Name:'Perry~')&sort=field(Name) asc

Above fuzzy search query is resulting following name as 'Ellysea Perry', 'Ellys Perry'

But why above query is not giving document having follwoing name 'Elly Perry' (as default distance parameter is 2 and 2 characters (se) are not present.)

Strings having editDistance as '2' , should come in output(Eg. 'Elly Perry').

I understand that "with max edit distance 2 i can have up to 2 insertions, deletions or substitutions."

Name available in loaded data - 'Elly Perry'
Input query parameter - (Name:'Ellyse~') AND (Name:'Perry~')

Since after deleting 2 characters from name 'Ellyse', It becomes 'Elly'. so it should result in output. Could someone help me find the missing piece? https://en.wikipedia.org/wiki/Levenshtein_distance

I expect the following row to match:

'Ellysea Perry',
'Ellys Perry', 
'Elly Perry'

But only get following two

'Ellysea Perry',
'Ellys Perry'

'Name' field is configured in managed-schema as follows-

<field name="Name" type="text_general" multiValued="false" indexed="true" stored="true" required="true"/>

FieldType 'text_general' description is as follows-

<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100" multiValued="true">
      <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
    </fieldType>

As per documentation ===> "To perform a fuzzy search, use the tilde ~ symbol at the end of a single-word term.".

References - https://solr.apache.org/guide/8_5/the-standard-query-parser.html#TheStandardQueryParser-FuzzySearches

try to be more clear, i'm in lack of ideas in this problem, even it sounds like a classic :I have spend hours trying to play around with this but have got nowhere.

I have resolved the error, Now it is working on edit-distance:2

Modified query is as follows :-

http://localhost:8983/solr/startsolr/select?indent=on&wt=json&q=(Name:Ellyse~) AND (Name:Perry~)&sort=field(Name) asc

But further - I have indexed 16 milion records in solr, but fuzzy search is not working for a specific token having 40K records. In rest of the cases its working.

Should i have to configure some parameters in solr-config.xml file?

0 Answers
Related