Applying Term, Filter and Aggregation in one Es Query using Nest

Question

I'm using ElasticSearch's Nest 7.17.x to try and applying a Term, Filter and Aggregation all on single query.

The Term is a document id restriction which should restrict all following filters and aggregations. The Filter is to restriction the price to a specific range. The aggregation to is to make price buckets.

In the example below, is looks like both the price filter and term restriction arent being applied as I'm being returned documents with ids outside of the restriction and with prices larger than the filter.

var orderChangedArgFilter = client.Search(s => s
    .Query(q => +q.Terms(p => p.Field("id").Terms(productIds)))
    .Aggregations(aggs => aggs
        .Filter("user filter with aggs", f => f
            .Filter(q => q.Range(rf => rf.Field("price").GreaterThanOrEquals(0.01).LessThan(50.0)))
            .Aggregations(childAggs => childAggs
                .Range("0 to 50 price agg", r => r.Field("price").Ranges(rs => rs.From(0.0).To(50.0)))
                .Range("50 to 100 price agg", r => r.Field("price").Ranges(rs => rs.From(50.0).To(100.0)))
                .Range("100 to 150 price agg", r => r.Field("price").Ranges(rs => rs.From(100.0).To(150.0)))
            )
        )
    )
);

How do I correct the query to have the Term restriction apply first and then the filter on top of it?

Edit 1:

It looks like the document Id term restriction is working as expected but the filter is not.

My engine was created via AppSearch. Looks like the datatype for the price field is actually text instead of number (on the abstracted elastic engine), even though I've specified number on the AppSearch engine. This seems to be the cause of the problem.

The raw index mappings

{
  ".ent-search-engine-documents-luke-test": {
    "mappings": {
      "dynamic": "true",
      "properties": {
        "price": {
          "fields": {
            "prefix": {
              "search_analyzer": "q_prefix",
              "type": "text",
              "analyzer": "i_prefix",
              "index_options": "docs"
            },
            "enum": {
              "ignore_above": 2048,
              "type": "keyword"
            },
            "float": {
              "ignore_malformed": true,
              "type": "double"
            },
            "joined": {
              "search_analyzer": "q_text_bigram",
              "type": "text",
              "analyzer": "i_text_bigram",
              "index_options": "freqs"
            },
            "stem": {
              "type": "text",
              "analyzer": "iq_text_stem"
            },
            "delimiter": {
              "type": "text",
              "analyzer": "iq_text_delimiter",
              "index_options": "freqs"
            },
            "location": {
              "ignore_malformed": true,
              "type": "geo_point",
              "ignore_z_value": false
            },
            "date": {
              "ignore_malformed": true,
              "type": "date",
              "format": "strict_date_time||strict_date"
            }
          },
          "type": "text",
          "analyzer": "iq_text_base",
          "index_options": "freqs"
        },
        "id": {
          "type": "keyword"
        },
        "title": {
          "fields": {
            "prefix": {
              "search_analyzer": "q_prefix",
              "type": "text",
              "analyzer": "i_prefix",
              "index_options": "docs"
            },
            "enum": {
              "ignore_above": 2048,
              "type": "keyword"
            },
            "float": {
              "ignore_malformed": true,
              "type": "double"
            },
            "joined": {
              "search_analyzer": "q_text_bigram",
              "type": "text",
              "analyzer": "i_text_bigram",
              "index_options": "freqs"
            },
            "stem": {
              "type": "text",
              "analyzer": "iq_text_stem"
            },
            "delimiter": {
              "type": "text",
              "analyzer": "iq_text_delimiter",
              "index_options": "freqs"
            },
            "location": {
              "ignore_malformed": true,
              "type": "geo_point",
              "ignore_z_value": false
            },
            "date": {
              "ignore_malformed": true,
              "type": "date",
              "format": "strict_date_time||strict_date"
            }
          },
          "type": "text",
          "analyzer": "iq_text_base",
          "index_options": "freqs"
        }
      },
      "dynamic_templates": [
        {
          "permissions": {
            "mapping": {
              "type": "keyword"
            },
            "match": "_*_permissions"
          }
        },
        {
          "thumbnails": {
            "mapping": {
              "type": "binary"
            },
            "match": "_thumbnail_*"
          }
        },
        {
          "data": {
            "match_mapping_type": "*",
            "mapping": {
              "fields": {
                "enum": {
                  "ignore_above": 2048,
                  "type": "keyword"
                },
                "float": {
                  "ignore_malformed": true,
                  "type": "double"
                },
                "delimiter": {
                  "type": "text",
                  "index_options": "freqs",
                  "analyzer": "iq_text_delimiter"
                },
                "joined": {
                  "search_analyzer": "q_text_bigram",
                  "type": "text",
                  "index_options": "freqs",
                  "analyzer": "i_text_bigram"
                },
                "prefix": {
                  "search_analyzer": "q_prefix",
                  "type": "text",
                  "index_options": "docs",
                  "analyzer": "i_prefix"
                },
                "location": {
                  "ignore_malformed": true,
                  "type": "geo_point",
                  "ignore_z_value": false
                },
                "date": {
                  "ignore_malformed": true,
                  "type": "date",
                  "format": "strict_date_time||strict_date"
                },
                "stem": {
                  "type": "text",
                  "analyzer": "iq_text_stem"
                }
              },
              "type": "text",
              "index_options": "freqs",
              "analyzer": "iq_text_base"
            }
          }
        }
      ]
    }
  }
}

Is it possible to still use the Nest client against an engine created via AppSearch?

userrig5 · Answer

My query had a couple of issues. Firstly I should have been using field name price.float which is the numeric representation of the price field on ES vs AppSearch. You could figure this out from the Explain endpoint offered by AppSearch.

The next issue was the structure of my query, I shouldn't have been using a FilteredAggregation. My intent was to not apply the filter specifically onto the aggregation, but to the query result.

Lastly, retrieving the results from the Aggregations was not that straight forward. Dumping the contents to console falsely shows an empty aggregation, however upon investigating the debugging info via the .EnableDebugMode() header, I was able to see that I was indeed receiving results. Applying a breakpoint and inspecting the results of the object allowed me to retrieve the object I wanted.

    var orderChangedArgFilter = client.Search(s => s
        .Query(q => +q.Range(rf => rf.Field("price.float").GreaterThanOrEquals(0.01).LessThan(50.0)) && +q.Terms(p => p.Field("id").Terms(productIds)))
        .Aggregations(a => a
            .Range("Price aggs", r => r
            .Field("price.float")
            .Ranges(rs => 
                rs.From(0.01).To(50.0).Key("0 to 50 price agg"),
                rs => rs.From(50.01).To(100).Key("50 to 100 price agg"),
                rs => rs.From(100.01).Key("From 100 price agg")))
        )
    );

    orderChangedArgFilter.Aggregations.Dump();
    // vs
    var agg = (Nest.BucketAggregate)orderChangedArgFilter.Aggregations.GetValueOrDefault("Price aggs");
    var buckets = agg.Items.Select(b => (RangeBucket)b).ToList();
    Console.WriteLine(buckets[0].Key + ", " + buckets[0].DocCount);

Note the line of text at the bottom of the below screen shot showing the Key and DocCount

Applying Term, Filter and Aggregation in one Es Query using Nest

1 Answers