I'd like to stream Cloudwatch logs from a specific group and log stream.
This command does a good job at streaming a group (including all the corresponding streams):
aws logs tail /aws/batch/job --follow --since 1d
I tried piping the result to grep and also specifying the --filter-pattern with the prefix of the desired stream but it simply returns nothing.
I've found the --format short parameter on aws logs tail is useful. It produces:
$ aws logs tail --follow /aws/lambda/do_something_neato --format short
2021-04-30T07:01:40 START RequestId: 8...1 Version: $LATEST
2021-04-30T07:01:40 info: Do something v1.0.0
2021-04-30T07:01:40 info: Doing some things(5 total).
2021-04-30T07:01:40 info: Doing thing 1...
2021-04-30T07:01:40 info: Doing thing 2...
That would otherwise be:
2021-04-30T07:01:40.683000+00:00 2021/04/30/[$LATEST]ce094d1cc6014a0da8c3df300aae4f36 START RequestId: 8...1 Version: $LATEST
2021-04-30T07:01:40.683000+00:00 2021/04/30/[$LATEST]ce094d1cc6014a0da8c3df300aae4f36 info: Do something v1.0.0
2021-04-30T07:01:40.683000+00:00 2021/04/30/[$LATEST]ce094d1cc6014a0da8c3df300aae4f36 info: Doing some things(5 total).
2021-04-30T07:01:40.683000+00:00 2021/04/30/[$LATEST]ce094d1cc6014a0da8c3df300aae4f36 info: Doing thing 1...
2021-04-30T07:01:40.683000+00:00 2021/04/30/[$LATEST]ce094d1cc6014a0da8c3df300aae4f36 info: Doing thing 2...
I initially went down the path of putting together an awk script to format things the way I wanted, only to find that piping the output of aws logs tail doesn't reliably work. But the above accomplishes what I want.
I Recommend to use cw .
With cw, you can use below command:
cw tail -f my-log-group:my-log-stream-prefix
--filter-pattern will only match the syntax of the log message so unless each stream was prefixed with the stream name you would not be able to match on pattern.
Can you you try piping to grep with the stream name in specifically i.e.
aws logs tail /aws/batch/job --follow --since 1d | grep "my\-stream\-name"
The aws logs tail command now has a --log-stream-name-prefix flag that you can use to only show logs from the desired log stream(s). However, you may need to update to the latest version in order to use it.
For example, you could probably do this in your case:
aws logs tail /aws/batch/job --follow --since 1d --log-stream-name-prefix my-stream-name
I'm actually really excited that they have added this feature. I've been wanting something like this as well for quite some time.