elasticsearch - filebeat - How to define multiline in filebeat.inputs with conditions?

Linux Dev 0
Viewed 27

in our cluster some apps are sending logs as multiline, and the problem is that the log structure is different from app to app.

How can we set up an 'if' condition that will include the

    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    multiline.negate: true
    multiline.match: after

In it?

Our code:

filebeatConfig: filebeat.yml: |

      filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
        - drop_event:
            when:
              contains:
                container.image.name: "kibana"
              

      output.logstash:
        hosts: ["logstash-listener:5044"]
filebeat elk logging elasticsearch
1 Answers

You need to use auto-discovery (either Docker or Kubernetes) with template conditions.

You will probably have at least two templates, one for capturing your containers that emit multiline messages and another for other containers.

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:               <--- your multiline condition goes here
            contains:
              kubernetes.namespace: xyz-namespace
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log
              multiline:
                pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
                negate: true
                match: after
              processors:
                - add_kubernetes_metadata:
                  host: ${NODE_NAME}
                  matchers:
                  - logs_path:
                     logs_path: "/var/log/containers/"
                - drop_event:
                    when:
                      contains:
                        container.image.name: "kibana"
user1Tjj0
Related