Can't disable field indexing on ElasticSearch AWS

Question

I'm using AWS ElasticSearch 7.9. I'm pushing logs to it with Filebeat 7.12 installed on an Elastic Beanstalk AMI2.

My log structure is as follow:

{
  "timestamp": "2021-04-07T22:58:08.012Z",
  "label": "My API",
  "level": "info",
  "module": "server",
  "message": "API is starting in production mode. Version 2.74.0",
  "metadata": {
    "foo": {
      "bar": "anything"
    }
  }
}

What I want is for ElasticSearch to stop indexing the metadata field as it can be litteraly anything. It is used for debug purpose when logging arbitrary objects.

I tried to update the templates (ecs ans filebeat) on ElasticSearch with the API by adding an explicit declaration for the field

"mappings" : {
    "_meta" : {
        "beat" : "filebeat",
        "version" : "7.12.0"
    },
    "properties" : {
        "metadata": {
            "type": "object",
            "enabled": false
        },
...

My filebeat configuration:

filebeat.inputs:
- type: log
  paths:
    - /var/log/web.stdout.log
  json.keys_under_root: true
  json.add_error_key: true

output.elasticsearch:
  hosts: ["https://somurl.es.amazonaws.com:443"]
  username: "username"
  password: "password"
  index: "my_api-filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"
setup.template.append_fields:
- name: metadata
  type: object
  enabled: false

setup.ilm.enabled: false

However the field metadata keeps getting indexed and I end up with a lot of pointless indexes and of course, conflicts, because some fields doesn't have the same type on two logs.

Is there something I'm not doing right ?

user_8160318 · Answer

You're on the right track but such an update call would be considered a breaking change raising the exception the [enabled] parameter can't be updated for the object mapping [metadata].

At the same time, individual mapping attributes cannot be removed so an effort to copy the old metadata into a new, temporary field, then remove the original field mapping, and then copy the temporary contents back to the original, yet updated metadata field would be futile…

The only option is thus to create a new index with the enabled: false attribute, reindex your logs into that new index, and reroute the upcoming log streams to the new index.

user_7167739 · Answer

Long story short: After some time away from it, I finally found what was causing the issue.

In my filebeat config I was using the index pattern my_api-filebeat-%{[agent.version]}-%{+yyyy.MM.dd} which was not recognized by elasticsearch, as the index pattern for the default templates is filebeat-*.

Just renaming my index pattern to filebeat-my_api-%{[agent.version]}-%{+yyyy.MM.dd} in my filebeat config file worked like a charm !

Can't disable field indexing on ElasticSearch AWS

2 Answers