Correct way to use modules in Filebeat

Viewed 1147

I'm slightly confused about the correct way to use Filebeat's modules, whilst running Filebeat in a Docker container. It appears that the Developers prefer the modules.d method, however it's not clear to me of their exact intentions.

Here is the relevant part of my filebeat.yml:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true
    reload.period: 60s

And the filebeat container definition from my docker-compose.yml:

  filebeat:
    build:
      context: filebeat/
      args:
        ELK_VERSION: $ELK_VERSION
    container_name: filebeat
    mem_limit: 2048m
    labels:
      co.elastic.logs/json.keys_under_root: true
      co.elastic.logs/json.add_error_key: true
      co.elastic.logs/json.overwrite_keys: true
    volumes:
      - type: bind
        source: ./filebeat/config/filebeat.docker.yml
        target: /usr/share/filebeat/filebeat.yml
        read_only: true
      - type: bind
        source: /volume1/@docker/containers
        target: /var/lib/docker/containers
        read_only: true
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
        read_only: true
      - type: bind
        source: /var/log
        target: /host-log
        read_only: true
      - type: volume
        source: filebeat-data
        target: /usr/share/filebeat/data
    user: root
    depends_on:
      - elasticsearch
      - kibana
    command: filebeat -e -strict.perms=false
    networks:
      - elk
    depends_on:
      - elasticsearch

With this configuration, I can docker exec into my container and activate modules (pulling in their default configuration), and setup pipelines and dashboards like so:

filebeat modules enable elasticsearch kibana system nginx
filebeat setup -e --pipelines

This all works fine, until I come to recreate my container, at which point the enabled modules are (unsurprisingly) disabled and I have to run this stuff again.

I tried to mount the modules.d directory on my local filesystem, expecting this to build the default modules.d config files (with their .disabled suffix) on my local filesystem, such that recreation of the container would persist the installed modules. To do so, the following was added as a mount to docker-compose.yml:

- type: bind
  source: ./filebeat/config/modules.d
  target: /usr/share/filebeat/modules.d
  read_only: false

When I do this, and recreate the container, it spins up with an empty modules.d directory and filebeat modules list returns no modules at all, meaning none can be enabled.

My current workaround is to copy each individual monitor's config file, and mount it specifically, like so:

- type: bind
  source: ./filebeat/config/modules.d/system.yml
  target: /usr/share/filebeat/modules.d/system.yml
  read_only: true
- type: bind
  source: ./filebeat/config/modules.d/nginx.yml
  target: /usr/share/filebeat/modules.d/nginx.yml
  read_only: true

This is suboptimal for a number of reasons, chiefly, if I want to enable a new module, and for it to persist my container being recreated, I need to:

  • docker exec into the container
  • get the default config file for the module I want to use
  • create a file on the local filesystem for the module
  • edit the docker-compose.yml file with the new bind mounted module config
  • recreate the container with docker-compose up --detach

The way I feel this should work is:

  • I mount modules.d to my local filesystem
  • I recreate the container
  • modules.d gets populated with all the default module config files
  • I enable modules by filebeat modules enable blah or by renaming the module config file from my local filesystem (removing the .disabled suffix)
  • Enabled modules and their config survive container recreation

One way around this could be to copy (urgh) the whole modules.d directory from a running container to my local filesystem and mounting that wholesale. That feels wrong too.

What am I misunderstanding or misconfiguring here? How are other people doing this?

1 Answers

I build a custom image for each type of beat, and embed the .yml configuration in my image. Then I use the filebeat.module property of the configuration file to setup my modules inside of that file.

I think the intention of using the modules.d folder approach is that it makes it easier to understand your module configuration for a filebeat instance that is working with multiple files.

That is my intention of this 1 image to 1 module/file type approach that I use for sure. All of my logic/configuration is stored along with the service that I am monitoring and not in one central, monolithic location.

Another benefit to this approach is each individual filebeat only has access to the log files it needs. In the case of collecting the docker logs like you are, you need to run in in privileged mode to bind mount to /var/run/docker.sock. If I want to run your compose file but do not want to run privileged (or if I am using windows and cannot) then I loose all the other monitoring that you have built out.

Related