Kibana scripted field which loops through an array

user_4640328 0
Viewed 2689

I am trying to use the metricbeat http module to monitor F5 pools.

I make a request to the f5 api and bring back json, which is saved to kibana. But the json contains an array of pool members and I want to count the number which are up.

The advice seems to be that this can be done with a scripted field. However, I can't get the script to retrieve the array. eg

doc['http.f5pools.items.monitor'].value.length()

returns in the preview results with the same 'Additional Field' added for comparison:

[
 {
  "_id": "rT7wdGsBXQSGm_pQoH6Y",
  "http": {
   "f5pools": {
    "items": [
     {
      "monitor": "default"
     },
     {
      "monitor": "default"
     }
    ]
   }
  },
  "pool.MemberCount": [
   7
  ]
 },

If I try

doc['http.f5pools.items']

Or similar I just get an error:

"reason": "No field found for [http.f5pools.items] in mapping with types []"

Googling suggests that the doc construct does not contain arrays?

  1. Is it possible to make a scripted field which can access the set of values? ie is my code or the way I'm indexing the data wrong.
  2. If not is there an alternative approach within metricbeats? I don't want to have to make a whole new api to do the calculation and add a separate field

-- update.

Weirdly it seems that the number values in the array do return the expected results. ie.

doc['http.f5pools.items.ratio']

returns

 {
  "_id": "BT6WdWsBXQSGm_pQBbCa",
  "pool.MemberCount": [
   1,
   1
  ]
 },

-- update 2

Ok, so if the strings in the field have different values then you get all the values. if they are the same you just get one. wtf?

metricbeat kibana elasticsearch
3 Answers

I'm adding another answer instead of deleting my previous one which is not the actual question but still may be helpful for someone else in future.

I found a hint in the same documentation:

Doc values are a columnar field value store

Upon googling this further I found this Doc Value Intro which says that the doc values are essentially "uninverted index" useful for operations like sorting; my hypotheses is while sorting you essentially dont want same values repeated and hence the data structure they use removes those duplicates. That still did not answer as to why it works different for string than number. Numbers are preserved but strings are filters into unique.

This “uninverted” structure is often called a “column-store” in other systems. Essentially, it stores all the values for a single field together in a single column of data, which makes it very efficient for operations like sorting.

In Elasticsearch, this column-store is known as doc values, and is enabled by default. Doc values are created at index-time: when a field is indexed, Elasticsearch adds the tokens to the inverted index for search. But it also extracts the terms and adds them to the columnar doc values.

Some more deep-dive into doc values revealed it a compression technique which actually de-deuplicates the values for efficient and memory-friendly operations.

Here's a NOTE given on the link above which answers the question:

You may be thinking "Well that’s great for numbers, but what about strings?" Strings are encoded similarly, with the help of an ordinal table. The strings are de-duplicated and sorted into a table, assigned an ID, and then those ID’s are used as numeric doc values. Which means strings enjoy many of the same compression benefits that numerics do.

The ordinal table itself has some compression tricks, such as using fixed, variable or prefix-encoded strings.

Also, if you dont want this behavior then you can disable doc-values

user_21963410
user_46403280

The answer for why doc doesnt work

Quoting below:

Doc values are a columnar field value store, enabled by default on all fields except for analyzed text fields.

Doc-values can only return "simple" field values like numbers, dates, geo- points, terms, etc, or arrays of these values if the field is multi-valued. It cannot return JSON objects

Also, important to add a null check as mentioned below:

Missing fields

The doc['field'] will throw an error if field is missing from the mappings. In painless, a check can first be done with doc.containsKey('field')* to guard accessing the doc map. Unfortunately, there is no way to check for the existence of the field in mappings in an expression script.

Also, here is why _source works

Quoting below:

The document _source, which is really just a special stored field, can be accessed using the _source.field_name syntax. The _source is loaded as a map-of-maps, so properties within object fields can be accessed as, for example, _source.name.first.

.

Responding to your comment with an example:

The kyeword here is: It cannot return JSON objects. The field doc['http.f5pools.items'] is a JSON object

Try running below and see the mapping it creates:

PUT t5/doc/2
{
   "items": [
     {
      "monitor": "default"
     },
     {
      "monitor": "default"
     }
    ]
}


GET t5/_mapping

{
  "t5" : {
    "mappings" : {
      "doc" : {
        "properties" : {
          "items" : {
            "properties" : {
              "monitor" : {  <-- monitor is a property of items property(Object)
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
user_21963410
Related