Solr wildcard search incorrect result

user_9545397 0
Viewed 505

I have some unexpected results when i make wildcard queries. I am using solr 6.6.0. edismax handler inside solr ui. The following query return results as expected without wildcard - firstNames:James, but when i add wildcard there are no results found. without wildcard with wildcard For firstNames field i use default fieldType text_en with default tokenizers and filters. When i run exact same query for firstNames:Stephen and firstNames:Stephen* i got results in both wildcard and not wildcard searches. Below is my field xml inside schema.xml:

  <field name="firstNames" type="text_en" multiValued="true" indexed="true" stored="true"/>

  <fieldType name="text_en" class="solr.TextField" positionIncrementGap="100">
    <analyzer type="index">
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.StopFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.EnglishPossessiveFilterFactory"/>
      <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
      <filter class="solr.PorterStemFilterFactory"/>
    </analyzer>
    <analyzer type="query">
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.SynonymFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/>
      <filter class="solr.StopFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.EnglishPossessiveFilterFactory"/>
      <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
      <filter class="solr.PorterStemFilterFactory"/>
    </analyzer>
  </fieldType>
solr tokenize java
3 Answers

When you're doing a wildcard query the analysis chain is not invoked (well, that's a small lie - it is, but only the components that are MultiTermAware - which usually means that the LowercaseFilter is the only thing that still is active).

Since you have a stemming filter and the possessive filter attached, the end s on James is removed. Since this only happens on index time (remember, when you're using a wildcard, the analysis chain is generally skipped on query), the token jame is stored in the index.

When you make the query firstNames:James*, you ask Solr to "find any document that contains tokens that start with James. Since what was stored is the token jame, there are no tokens matching james.

When you try this with Stephen instead, neither stemming or possessive filter removes the end of the word, so Stephen* looks for any token starting with stephen, and since that token is present (nothing got changed), a match is returned.

The solution depends on your use case; there is no need for a stemming or possessive filter on a name field, since that doesn't really make sense for names (instead you might apply your own logic to match similar-ish names). Another option is to use an ngramfilter instead, effectively generating a token for each prefix and infix version of the token (foo, f, fo, oo, o).

user_1376500
  1. Do not remove stop words. That is a space-saving hack from the 1970s. It makes some words unsearchable, so queries like "vitamin a" will never work because "a" is a stop word. Here is a blog post listing movie titles which are 100% stop words.

https://observer.wunderwood.org/2007/05/31/do-all-stopword-queries-matter/

  1. Do not use wildcards with stemming. That will show matches on the stem, not the surface word. You want a separate field with just the lowercase filter.

  2. Do not use stemming on personal names. You do not want to stem "Steve Jobs" to "steve job" or "william golding" to "william gold", for example.

  3. Even better, use the ICU Folding Filter instead of just lowercasing.

https://lucene.apache.org/solr/guide/8_7/filter-descriptions.html#icu-folding-filter

user_8956140

About stopwords, the response to the question "do I have to use stopwords" is not "yes" or "no". It is "why not" but intelligently according what your datas are. For a drug database, "a", "b", "c" ... shouldn't be in the stopwords definition file. For a movie titles which are 100% stop words database, the title field must not use stopwords, but maybe the description field should.

user_59989150
Related