Is there a way to find the 10 documents that are "nearest" search values in Mongo using Aggregation Pipeline?

B Seven 0
Viewed 16

This is not for geospatial data.

Each document has up to 10 integer values:

{ val1: 5,
  val2: 8,
  val3: -4,
  ...

How to find the 10 documents "nearest" { val1: 4, val2: -1, ... ?

I can see using $addField to create a distance for each document

{ $addFields: { distance: ($val1-4)^2 + ($val2+1)^2 ...

And then sorting and $limit...

But I'm not sure if this will be performant (although there are only 40K documents)...

Perhaps there is a better way?

aggregation-framework mongodb
1 Answers

Without any preprocess there is no better way, If you have some prior knowledge on the data distribution you could add rigid rules to filter most documents on the initial match ( like val1: {$lt: 11, $gt: 0}), however if this returns less than 10 results you'll have to query again.

This is a very common access pattern that already has some production grade solutions, I recommend you choose one of those unless you want to develop something to custom fit your needs.

  1. Use a database that is built for vector querying, for example elasticsearch has the vector type, once the data is indexed it gives you OOB search engine functionality with various distance formula's.

  2. Allow lower accuracy in your queries, this combined with some preprocess approaches can help querying time, Here is a very interesting post on how spotify dealt with this issue. The basically split their data into different clusters and then each query does not need to scan the entire dataset. Again this compromises query accuracy.

Tom Slabbaert0
Related