Sort by constant number

Question

I need to randomize Solr (6.6.2) search results, but the order needs to be consistent given a specific seed. This is for a paginated search that returns a limited result set from a much larger one, so I must do the ordering at the query level and not at the application level once the data has been fetched.

Initially I tried this:

https://localhost:8984/solr/some_index/select?q=*:*&sort=random_999+ASC

Where 999 is a constant that is fed in when constructing the query prior to sending it to Solr. The constant value changes for each new search.

This solution works. However, when I run the query a few times, or run it on different Solr instances, the ordering is different.

After doing some reading, random_ generates a number via:

fieldName.hashCode() + context.docBase + (int)top.getVersion()

This means that when the random number is generated, it takes the index version into account. This becomes problematic when using a distributed architecture or when indexes are updated, as is well explained here.

There are various recommended solutions online, but I am trying to avoid writing a custom random override. Is there some type of trick where I can feed in some type of function or equation to the sort param?

For example:

min(999,random_999)

Though this always results in the same order, even when either of the values change.

This question is somewhat similar to this other question, but not quite.

I searched for answers on SO containing solr.RandomSortField, and while they point out what the issue is, none of them have a solution. It seems the best way would be to override the solr.RandomSortField logic, but it's not clear how.

Prior Research

user_4015994 · Answer

Even after implementing a custom random sort field, the results still differed across instances of Solr.

I ended up adding a new field that is populated at index time which is a 32 bit hash of an ID field that already existed in the document.

I then built a "stateless" linear congruential generator to produce a set of acceptably random numbers to use for sorting:

?sort=mod(product(hash_int_id,{seedConstant},982451653), 104395301) asc

Since this function technically passes a new seed for each row, and because it does not store state (like rand.Next() would), this solution is admittedly inferior and it is not a true PRNG; however, it does seem to get me most of the way there. Note that you will have to tune your values depending on the size of your data set and the size of the values in your hash_int_id equivalent field.

Sort by constant number

1 Answers