How to Send Kubernetes Logs to AWS CloudWatch?

Question

AWS CloudWatch Logs in Docker

Setting an AWS CloudWatch Logs driver in docker is done with log-driver=awslogs and log-opt, for example -

#!/bin/bash

docker run \
    --log-driver=awslogs \
    --log-opt awslogs-region=eu-central-1 \
    --log-opt awslogs-group=whatever-group \
    --log-opt awslogs-stream=whatever-stream \
    --log-opt awslogs-create-group=true \
    wernight/funbox \
        fortune

My Problem

I would like to use AWS CloudWatch logs in a Kubernetes cluster, where each pod contains a few Docker containers. Each deployment would have a separate Log Group, and each container would have a separate stream. I could not find a way to send the logging parameters to the docker containers via Kubernetes create / apply.

My Question

How can I send the log-driver and log-opt parameters to a Docker container in a pod / deployment?

What have I tried

Setting relevant parameters for the Docker daemon on each machine. It's possible, but this way all the containers on the same machine would share the same stream - therefore irrelevant for my case.
RTFM for kubectl apply
Reading the relevant README in kops
Read Kubernetes Logging Architecture

user_692076 · Answer

From what I understand, Kubernetes prefer Cluster-level logging to Docker logging driver.

We could use fluentd to collect, transform, and push container logs to CloudWatch Logs.

All you need is to create a fluentd DaemonSet with ConfigMap and Secret. Files can be found in Github. It has been tested with Kubernetes v1.7.5.

The following are some explains.

In

With DaemonSet, fluentd collect every container logs from the host folder /var/lib/docker/containers.

Filter

fluent-plugin-kubernetes_metadata_filter plugin load the pod's metadata from Kubernetes API server.

The log record would be like this.

{
    "log": "INFO: 2017/10/02 06:44:13.214543 Discovered remote MAC 62:a1:3d:f6:eb:65 at 62:a1:3d:f6:eb:65(kube-235)
",
    "stream": "stderr",
    "docker": {
        "container_id": "5b15e87886a7ca5f7ebc73a15aa9091c9c0f880ee2974515749e16710367462c"
    },
    "kubernetes": {
        "container_name": "weave",
        "namespace_name": "kube-system",
        "pod_name": "weave-net-4n4kc",
        "pod_id": "ac4bdfc1-9dc0-11e7-8b62-005056b549b6",
        "labels": {
            "controller-revision-hash": "2720543195",
            "name": "weave-net",
            "pod-template-generation": "1"
        },
        "host": "kube-234",
        "master_url": "https://10.96.0.1:443/api"
    }
}

Make some tags with Fluentd record_transformer filter plugin.

{
    "log": "...",
    "stream": "stderr",
    "docker": {
        ...
    },
    "kubernetes": {
        ...
    },
    "pod_name": "weave-net-4n4kc",
    "container_name": "weave"
}

Out

fluent-plugin-cloudwatch-logs plugin send to AWS CloudWatch Logs.

With log_group_name_key and log_stream_name_key configuration, log group and stream name can be any field of the record.


  @type cloudwatch_logs
  log_group_name_key pod_name
  log_stream_name_key container_name
  auto_create_stream true
  put_log_events_retry_limit 20

user_7262569 · Answer

As per kubernate, Kubernetes provides no native storage solution for log data, but you can integrate many existing logging solutions into your Kubernetes cluster and kubernate cluster-level-logging-architectures.

Kubernetes doesn’t specify a logging agent, but two optional logging agents are packaged with the Kubernetes release: Stackdriver Logging for use with Google Cloud Platform, and Elasticsearch. You can find more information and instructions in the dedicated documents. Both use fluentd with custom configuration as an agent on the node.

Fluentd image to send Kubernetes logs to CloudWatch too, so you can use that to Deploy,

user_2707671 · Answer

You could use a Helm chart to install Fluentd:

$ helm install --name my-release incubator/fluentd-cloudwatch

This is from: https://github.com/kubernetes/charts/tree/master/incubator/fluentd-cloudwatch

user_8280034 · Answer

Sliverfox has a great answer. You don't have to build your own image. Could also directly use fluentd official docker image, fluent/fluentd-kubernetes-daemonset:cloudwatch. The code is on fluentd-kubernetes-daemonset github.

You could replace the default fluent.conf with the configmap. Like below in the ds.yaml, and write your own fluent.conf in configmap.yaml. For the complete yaml files, you could refer to the example ds.yaml and configmap.yaml that we wrote.

    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: varlibdockercontainers
      mountPath: /var/lib/docker/containers
      readOnly: true
    - name: config-volume
      mountPath: /fluentd/etc/
  volumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers
  - name: config-volume
    configMap:
      name: fluentd-cw-config