Python Whoosh - Combining Results

user_8729568 0
Viewed 929

Thanks for taking the time to answer this in advance. I'm relatively new to both Python (3.6) and Whoosh (2.7.4), so forgive me if I'm missing something obvious.

Whoosh 2.7.4 — Combining Results Error

I'm trying to follow the instructions in the Whoosh Documentation here on How to Search > Combining Results. However, I'm really lost in this section:

# Get the terms searched for
termset = set()
userquery.existing_terms(termset)

As I run my code, it produces this error:

'set' object has no attribute 'schema'

What went wrong?

I also looked into the docs about the Whoosh API on this, but I just got more confused about the role of ixreader. (Or is it index.Index.reader()?) Shrugs

A Peek at My Code

Schema

schema = Schema(uid=ID(unique=True, stored=True),  # unique ID
                indice=ID(stored=True, sortable=True),
                title=TEXT,
                author=TEXT,
                body=TEXT(analyzer=LanguageAnalyzer(lang)),
                hashtag=KEYWORD(lowercase=True, commas=True,
                                scorable=True)
                )

The relevant fieldnames are the 'hashtag' and 'body'. Hashtags are user selected keywords for each document, and body is the text in the document. Pretty self-explanatory, no?

Search Function

Much of this is lifted directly from Whoosh Doc. Note, dic is just a dictionary containing the query string. Also, it should be noted that the error occurs during userquery.existing_terms(termset), so if the rest of it is bunk, my apologies, I haven't gotten that far.

try:
            ix = index.open_dir(self.w_path, indexname=lang)
            qp = QueryParser('body', schema=ix.schema)
            userquery = qp.parse(dic['string'])
            termset = set()
            userquery.existing_terms(termset)
            bbq = Or([Term('hashtag', text) for fieldname, text
                      in termset if fieldname == 'body'])
            s = ix.searcher()
            results = s.search(bbq, limit=5)
            allresults = s.search(userquery, limit=10)
            results.upgrade_and_extend(allresults)
            for r in results:
                print(r)
except Exception as e:
            print('failed to search')
            print(e)
            return False
finally:
            s.close()

Goal of My Code

I am taking pages from different files (pdf, epub, etc) and storing each page's text as a separate 'document' in a whoosh index (i.e. the field 'body'). Each 'document' is also labeled with a unique ID (uid) that allows me to take the search Results and determine the pdf file from which it comes and which pages contain the search hit (e.g. the document from page 2 of "1.pdf" has the uid 1.2). In other words, I want to give the user a list of page numbers that contain the search term and perhaps the pages with the most hits. For each file, the only document that has hashtags (or keywords) is the document with a uid ending in zero (i.e. page zero, e.g. uid 1.0 for "1.pdf"). Page zero may or may not have a 'body' too (e.g. the publish date, author names, summary, etc). I did this in order to prevent one document with more pages to be dramatically ranked higher from another with considerably less pages because of the multiple repetitions of the keyword over each 'document' (i.e. page).

Ultimately, I just want the code to elevate documents with the hashtag over documents with just search hits in the body text. I thought about just boosting the hashtag field instead, but I'm not sure what the mechanics of that is and the documentation recommends against this.

Suggestions and corrections would be greatly appreciated. Thank you again!

whoosh python-3.x python
1 Answers

The code from your link doesn't look right to me. It too gives me the same error. Try rearranging your code as follows:

try:
            ix = index.open_dir(self.w_path, indexname=lang)
            qp = QueryParser('body', schema=ix.schema)
            userquery = qp.parse(dic['string'])
            s = ix.searcher()
            allresults = s.search(userquery, limit=10)
            termset = userquery.existing_terms(s.reader())
            bbq = Or([Term('hashtag', text) for fieldname, text in termset if fieldname == 'body'])
            results = s.search(bbq, limit=5)

            results.upgrade_and_extend(allresults)
            for r in results:
                print(r)
except Exception as e:
            print('failed to search')
            print(e)
            return False
finally:
            s.close()

existing_terms requires a reader so I create the searcher first and give its reader to it.

As for boosting a field, the mechanics are quite simple:

schema = Schema(title=TEXT(field_boost=2.0), body=TEXT).

Add a sufficiently high boost to bring hashtag documents to the top and be sure to apply a single query on both body and hashtag fields.

Deciding between boosting or combining depends on whether you want all matching hashtag documents to be always, absolutely at the top before any other matches show. If so, combine. If instead you prefer to strike a balance in relevance albeit with a stronger bias for hashtags, boost.

user_42375910
Related