restructure elasticsearch index to allow filtering on sum of values

user_145501 0
Viewed 161

I've an index of products.

Each product, has several variants (can be a few or hundreds, each has a color & size e.g. Red)

Each variant, is available (in a certain quantity) at several warehouses (aronud 100 warehouses).

Warehouses have codes e.g. AB, XY, CD etc.

If I had my choice, I'd index it as:

stock: {
  Red: {
    S: { AB: 100, XY: 200, CD: 20 },
    M: { AB: 0, XY: 500, CD: 20 },
    2XL: { AB: 5, XY: 0, CD: 9 }
  },
  Blue: { 
    ...
  }
}

Here's a kind of customer query I might receive:

Show me all products, that have Red.S color in stock (minimum 100) at warehouses AB & XY.

So this would probably be a filter like

Red.S.AB > 100 AND Red.S.XY > 100

I'm not writing whole filter query here, but its straightforward in elastic.

We might also get SUM queries, e.g. the sum of inventories at AB & XY should be > 500.

That'd be easy through a script filter, say Red.S.AB + Red.S.XY > 500

The problem is, given 100 warehouses, 100 sizes, 25 colors, this easily needs 100*100*25 = 250k mappings. Elasticsearch simply can't handle that many number of keys.

The easy answer is use nested documents, but nested documents pose a particular problem. We cannot sum across a given selection of nested documents, and nested docs are slow, specially when we're going to have 250k per product.

I'm open to external solutions than elastic as well. We're rails/postgres stack.

elasticsearch
2 Answers

You have your product index with variants, that's fine, but I'd use another index for managing anything related to the multi-warehouse stock. One document per product/size/color/warehouse with the related count. For instance:

{
  "product": 123,
  "color": "Red",
  "size": "S",
  "warehouse": "AB",
  "quantity": 100
}

{
  "product": 123,
  "color": "Red",
  "size": "S",
  "warehouse": "XY",
  "quantity": 200
}

{
  "product": 123,
  "color": "Red",
  "size": "S",
  "warehouse": "CD",
  "quantity": 20
}
etc...

That way, you'll be much more flexible with your stock queries, because all you'll need is to filter on the fields (product, color, size, warehouse) and simply aggregate on the quantity field, sums, averages or whatever you might think of.

You will probably need to leverage the bucket_script pipeline aggregation in order to decide whether sums are above or below a desired threshold.

It's also much easier to maintain the stock movements by simply indexing the new quantity for any given combination than having to update the master product document every time an item gets out of the stock.

No script, no nested documents required.

user_46045790

The best possible solution will be to create separate indexes for the warehouses and each warehouse index will have documents. One document per product/size/color/warehouse with related values like this:

{
  "product": 123,
  "color": "Red",
  "size": "S",
  "warehouse": "AB",
  "quantity": 100
}

This will reduce your mappings 100 * 25 = 2500 mappings per index.

Rest for other operations, I feel @Val has mentioned in his answer which is quite impressive and beautiful.

Coming to external solutions, I would say you want to carry to out task of storing data, searching it and fetching it. Elasticsearch and Apache Solr are the best search engines to carry out these kind of tasks. I have not tried Apache Solr but I would highly recommend to go with Elasticsearch because of it's features, active community support and searching is really fast. Searching can also be made fast using analyzers and tokenizers. It also has some features like Full-Text Searching and Term Level Searching to customize searching according to situation or problem statement.

user_72993400
Related