How do you send Fargate EKS Fluent Bit Logs from different services to separate Cloudwatch groups?

user_12458182 0
Viewed 1643

I have followed this guide to configure Fluent Bit and Cloudwatch on my EKS cluster, but currently all of the logs go to one log group. I tried to follow a separate tutorial that used a kubernetes plugin for Fluent Bit to tag the services before the reached the [OUTPUT] configuration. This caused issues because Fargate EKS currently does not handle Fluent Bit [INPUT] configurations as per the bottom of this doc.

Has anyone encountered this before? I'd like to split the logs up into separate services.

Here is my current YAML file .. I added the parser and filter to see if I could gain any additional information to work with over on Cloudwatch.

kind: Namespace
apiVersion: v1
metadata:
  name: aws-observability
  labels:
    aws-observability: enabled
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  parsers.conf: |
    [PARSER]
        Name docker
        Format json
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep On
        
  filters.conf: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
        # Kube_Tag_Prefix kube.var.log.containers.
        Kube_URL https://kubernetes.default.svc:443
        Merge_Log On
        Merge_Log_Key log_processed
        Use_Kubelet true
        Buffer_Size 0
        Dummy_Meta true
  
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match   *
        region us-east-1
        log_group_name fluent-bit-cloudwatch2
        log_stream_prefix from-fluent-bit-
        auto_create_group On
amazon-eks amazon-cloudwatch aws-fargate fluent-bit kubernetes
1 Answers

So I found out that it is actually simple to do this.

The default tag of input on fluent bit contains the name of the service you are logging from, so you can actually stack multiple [OUTPUT] blocks each using the wildcard operator around the name of your service . That was all I had to do to get the streams to get sent to different log groups. Here is my YAML for reference.

kind: Namespace
apiVersion: v1
metadata:
  name: aws-observability
  labels:
    aws-observability: enabled
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match   *logger*
        region us-east-1
        log_group_name logger-fluent-bit-cloudwatch
        log_stream_prefix from-fluent-bit-
        auto_create_group On
        
    [OUTPUT]
        Name cloudwatch_logs
        Match   *alb*
        region us-east-1
        log_group_name alb-fluent-bit-cloudwatch
        log_stream_prefix from-fluent-bit-
        auto_create_group On
user_124581820
Related