Error extracting container id - source value does not contain matcher's logs_path '/var/lib/docker/containers/'

user_2628868 0
Viewed 4596

I am collect container's log using filebeat in kubernetes cluster, and now collected log shows this error:

2020-06-10T09:26:35.831Z ERROR [kubernetes] add_kubernetes_metadata/matchers.go:91 Error extracting container id - source value does not contain matcher's logs_path '/var/lib/docker/containers/'.

this is the full log output:

enter image description here

I find the filebeat was listening is the node meowk8sslave2 and login into this node found the path exists. why the error could happen? this is my filebeat config:

{
    "filebeat.yml": "filebeat.inputs:
        - type: container
          paths:
            - /var/log/containers/*.log
          processors:
          - add_kubernetes_metadata:
              host: ${NODE_NAME}
              matchers:
              - logs_path:
                  logs_path: \"/var/log/containers/\"

        output.elasticsearch:
          host: '${NODE_NAME}'
          hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
        "
}
kubernetes
3 Answers

Look inside your filebeat pod where exactly the logs are made available.

I was testing the ELK stack on Minikube.

In my case it was inside /var/lib/docker/containers/*/*.log

So this one worked for me.

filebeatConfig:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths:
        - /var/lib/docker/containers/*/*.log
      processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
          - logs_path:
              logs_path: "/var/lib/docker/containers/"
    output.elasticsearch:
      host: '${NODE_NAME}'
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
user_62487900

change 

filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    # filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      node: ${NODE_NAME}
    #      hints.enabled: true
    #      hints.default_config:
    #        type: container
    #        paths:
    #          - /var/log/containers/*${data.kubernetes.container.id}.log

to 

# filebeat.inputs:
    # - type: container
    #   paths:
    #     - /var/log/containers/*.log
    #   processors:
    #     - add_kubernetes_metadata:
    #         host: ${NODE_NAME}
    #         matchers:
    #         - logs_path:
    #             logs_path: "/var/log/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: container
           paths:
             - /var/log/containers/*${data.kubernetes.container.id}.log

Reference: https://discuss.elastic.co/t/problem-to-update-to-filebeat-7-7-0-and-parser-nginx-ingress-controller-on-kubernetes/232461/2

user_2616830

works for me:
update processors
from:

processors:
  - add_cloud_metadata: ~
  - add_kubernetes_metadata:
        in_cluster: true
  - drop_event.when.regexp.message: "kube-probe"

to:

processors:
  - add_cloud_metadata: ~
  - add_kubernetes_metadata:
        in_cluster: true
        host: ${NODE_NAME}
        matchers:
        - logs_path:
              logs_path: "/var/log/containers/"
  - drop_event.when.regexp.message: "kube-probe"

maybe you need to update your nginx module log path to:
/var/log/containers/*-${data.kubernetes.container.id}.log

user_94859290
Related