I'm slightly confused about the correct way to use Filebeat's modules, whilst running Filebeat in a Docker container. It appears that the Developers prefer the modules.d method, however it's not clear to me of their exact intentions.
Here is the relevant part of my filebeat.yml:
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 60s
And the filebeat container definition from my docker-compose.yml:
filebeat:
build:
context: filebeat/
args:
ELK_VERSION: $ELK_VERSION
container_name: filebeat
mem_limit: 2048m
labels:
co.elastic.logs/json.keys_under_root: true
co.elastic.logs/json.add_error_key: true
co.elastic.logs/json.overwrite_keys: true
volumes:
- type: bind
source: ./filebeat/config/filebeat.docker.yml
target: /usr/share/filebeat/filebeat.yml
read_only: true
- type: bind
source: /volume1/@docker/containers
target: /var/lib/docker/containers
read_only: true
- type: bind
source: /var/run/docker.sock
target: /var/run/docker.sock
read_only: true
- type: bind
source: /var/log
target: /host-log
read_only: true
- type: volume
source: filebeat-data
target: /usr/share/filebeat/data
user: root
depends_on:
- elasticsearch
- kibana
command: filebeat -e -strict.perms=false
networks:
- elk
depends_on:
- elasticsearch
With this configuration, I can docker exec into my container and activate modules (pulling in their default configuration), and setup pipelines and dashboards like so:
filebeat modules enable elasticsearch kibana system nginx
filebeat setup -e --pipelines
This all works fine, until I come to recreate my container, at which point the enabled modules are (unsurprisingly) disabled and I have to run this stuff again.
I tried to mount the modules.d directory on my local filesystem, expecting this to build the default modules.d config files (with their .disabled suffix) on my local filesystem, such that recreation of the container would persist the installed modules. To do so, the following was added as a mount to docker-compose.yml:
- type: bind
source: ./filebeat/config/modules.d
target: /usr/share/filebeat/modules.d
read_only: false
When I do this, and recreate the container, it spins up with an empty modules.d directory and filebeat modules list returns no modules at all, meaning none can be enabled.
My current workaround is to copy each individual monitor's config file, and mount it specifically, like so:
- type: bind
source: ./filebeat/config/modules.d/system.yml
target: /usr/share/filebeat/modules.d/system.yml
read_only: true
- type: bind
source: ./filebeat/config/modules.d/nginx.yml
target: /usr/share/filebeat/modules.d/nginx.yml
read_only: true
This is suboptimal for a number of reasons, chiefly, if I want to enable a new module, and for it to persist my container being recreated, I need to:
docker execinto the container- get the default config file for the module I want to use
- create a file on the local filesystem for the module
- edit the
docker-compose.ymlfile with the new bind mounted module config - recreate the container with
docker-compose up --detach
The way I feel this should work is:
- I mount modules.d to my local filesystem
- I recreate the container
- modules.d gets populated with all the default module config files
- I enable modules by
filebeat modules enable blahor by renaming the module config file from my local filesystem (removing the.disabledsuffix) - Enabled modules and their config survive container recreation
One way around this could be to copy (urgh) the whole modules.d directory from a running container to my local filesystem and mounting that wholesale. That feels wrong too.
What am I misunderstanding or misconfiguring here? How are other people doing this?