AWS CloudWatch Logs Stream - how configure awslogs to write every day new log stream to the same log group from the same instance?

user_14537223 0
Viewed 3205

I have 1 instance that use the following awslogs configuration file:

[general]
state_file = /var/lib/awslogs/agent-state

[logstream-0]
log_stream_name = controller
log_group_name = robots/controller
file = /mnt/data/log/controller.log
initial_position = start_of_file

[logstream-1]
log_stream_name = catcher
log_group_name = robots/catcher
file = /mnt/data/log/catcher.log
initial_position = start_of_file

CloudWatch now holds 2 log groups (each of them have one log stream inside):

  1. robots/controller:

    log stream :

    controller

  2. robots/catcher:

    log stream :

    catcher

My goal here is to create a new log stream everyday (by timestamp of course).

Should look like this (Datetime format is not really matter):

  1. robots/controller:

    log streams :

    controller_2020/12/24

controller_2020/12/23

controller_2020/12/22

  2. robots/catcher:

    log streams :

    catcher_2020/12/24

catcher_2020/12/23

catcher_2020/12/22

Any idea?

amazon-cloudwatch devops amazon-web-services logging
3 Answers

For log file segregation, File option can point to a specific file or multiple files using wildcards such as /var/log/system.log*). Only the latest file is pushed to CloudWatch Logs based on file modification time. you need to write logs to a new file that matches the pattern, you need to configure your source logs to create a new file daily

For time stamp use: datetime_format = %Y-%m-%d %H:%M:%S

user_89059090

Here is a workaround. This works by modifying the systemd service file and adding a script that edits place holders in the cw agent config.

/opt/aws/amazon-cloudwatch-agent/etc/config-template.json:

{
    "agent": {
        "metrics_collection_interval": 10,
        "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
        "region": "us-east-1"
    },
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [{
                    "file_path": "/var/log/messages",
                    "log_group_name": "/hybrid_instance_1",
                    "log_stream_name": "%%DATE%%-messages",
                    "timezone": "UTC"
                }]
            }
        },
        "force_flush_interval": 15
    }
}

/etc/systemd/system/amazon-cloudwatch-agent.service:

[Unit]
Description=Amazon CloudWatch Agent
After=network.target

[Service]
Type=simple
ExecStartPre=-/bin/bash -c "/usr/sbin/update_cw_agent_config"
ExecStart=/opt/aws/amazon-cloudwatch-agent/bin/start-amazon-cloudwatch-agent
KillMode=process
Restart=on-failure
RestartSec=60s

[Install]
WantedBy=multi-user.target

/usr/sbin/update_cw_config:

#!/bin/bash
DATE_STRING=$(date +"%Y\/%m\/%d")
sed -r "s/%%DATE%%/${DATE_STRING}/g" /opt/aws/amazon-cloudwatch-agent/etc/config-template.json > /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

You can then run /usr/sbin/update_cw_config by restarting the service via crontab. You could of course not edit the service file and just run the script every day at midnight with "systemctl restart amazon-cloudwatch-agent" appended. I am using the systemd file due to wanting a new stream with the date + minutes and seconds added anytime the agent restarts, not just by a cron.

user_55323020

They don't say anything in their documentation, but looking at their codebase there is a placeholder called {date}

It's formatting in a different date format, but at least it's there without hacky workarounds. Tested it in our logging and it resolves to the format 2021-12-21.

user_177308160
Related