I'm trying to upload log files to AWS CloudWatch. The application is outputting log4j2 style JSON into a file:
https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout
AWS provide 2 cloudwatch log agents for this task. An 'older' agent, and a 'unified' agent:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_GettingStarted.html
I have tried using both agents, but run into the following problems, mainly related to parsing timestamps and the fact that the agent is doing a regex match on the entire log line and is not parsing the it as JSON:
An example log message (additional fields omitted):
{"message": "Processing data for previous day: 2019-06-17T02:01:00", "timestamp": "2019-06-18T17:16:19.338000+0100"}
- The older agent threw an exception because it was attempting to use my configured timestamp format to parse the timestamp in the message, not the one in the
timestampfield. - The unified agent is unable to parse timestamps with sub-second precision which results in problems when attempting to combine log streams from multiple sources.
So, is there a better tool/strategy to upload JSON formatted log4j2 files to CoudWatch?