fluent bit kubernetes annotation does not work

Question

I deploy fluent bit to kubernetes. And I deploy one pod with annotation fluentbit.io/parser: cri. But it still parse the log with the parser ivyxjc which is config in INPUT.

fluent-bit config

apiVersion: v1
kind: ConfigMap
metadata:
    name: fluent-bit-config
    labels:
        k8s-app: fluent-bit
data:
    # Configuration files: server, input, filters and output
    # ======================================================
    fluent-bit.conf: |
        [SERVICE]
            Flush         1
            Log_Level     info
            Daemon        off
            Parsers_File  parsers.conf
            HTTP_Server   On
            HTTP_Listen   0.0.0.0
            HTTP_Port     2020

        @INCLUDE input-kubernetes.conf
        @INCLUDE filter-kubernetes.conf
        @INCLUDE output-elasticsearch.conf

    input-kubernetes.conf: |
        [INPUT]
            Name              tail
            Tag               kube.*
            Path              /var/log/containers/*.log
            Parser            ivyxjc
            DB                /var/log/flb_kube.db
            Mem_Buf_Limit     200MB
            Skip_Long_Lines   On
            Refresh_Interval  10

    filter-kubernetes.conf: |
        [FILTER]
            Name                kubernetes
            Match               kube.*
            Kube_URL            https://kubernetes.default.svc:443
            Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
            Kube_Tag_Prefix     kube.var.log.containers.
            Merge_Log           On
            k8s-logging.parser  On
            K8S-Logging.Exclude On

    output-elasticsearch.conf: |
        [OUTPUT]
            Name            es
            Match           *
            Host            ${FLUENT_ELASTICSEARCH_HOST}
            Port            ${FLUENT_ELASTICSEARCH_PORT}
            HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
            HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWD}
            Logstash_Format On
            Replace_Dots    On
            Retry_Limit     False
            tls             On
            tls.verify      Off

    parsers.conf: |
        [PARSER]
            # http://rubular.com/r/tjUt3Awgg4
            Name cri
            Format regex
            Regex ^(?[^ ]+) (?stdout|stderr) (?[^ ]*) (?.*)$
            Time_Key    time
            Time_Format %Y-%m-%dT%H:%M:%S.%L%z

        [PARSER]
            Name ivyxjc
            Format regex
            Regex ^(?[^ ]+) (?stdout|stderr) (?[^ ]*) (?.*)$
            Time_Key    time
            Time_Format %Y-%m-%dT%H:%M:%S.%L%z

Pod description:

...
Name:         logger-5c6658b5dd-66zkw
Namespace:    logger
Priority:     0
Start Time:   Fri, 15 Oct 2021 15:28:47 +0800
Labels:       app=logger
              pod-template-hash=5c6658b5dd
Annotations:  fluentbit.io/parser: cri
              fluentbit.io/parser_stderr: cri
              fluentbit.io/parser_stdout: cri
Status:       Running
...

user_14808928 · Answer

We were having this issue and ended up finding a mis-configuration in the cluster role binding for fluent. There was a type in the name of our service account and that issue caused really weird behaviour in fluent including lines not being parsed with the correct parser and many tags not being applied but there were no obvious log lines indicating anything had gone wrong even at "debug" log level.

user_13694129 · Answer

k8s-logging.parser

This option tells fluent bit agent to use parser from the annotation that will be used for the "log" keyword. The INPUT parser will be applied as per usual. Afterwards "KUBERNETES" filter picks up the input and then the parser dictated by "fluentbit.io/parser: parser_name_here" will pick up values from the "log" keyword.

Reference to docs.

Unfortunately that does not work in some cases as well, I have not been able to chase it down yet, but will update this answer if I do find something.

fluent bit kubernetes annotation does not work

2 Answers