How to count number of occurrences of IP address in CloudWatch log

user_1408347 0
Viewed 4052

Let's say I have a cloud-watch log that looks like this:

enter image description here

The IP on the most left-hand side represents where the request comes from.

Is it possible to list the number of occurrences of each IP in a time range that I specified?

I mean I want to create data which look like this:

time range: 2019-06-02 00:00:00 - 2019-06-04 13:00:00
number of occurrences of `172.31.13.80`: 130
number of occurrences of `172.31.25.110`: 112
number of occurrences of `172.31.8.124`: 99
number of occurrences of `172.31.8.121`: 86

It seems that CloudWatch Logs Insights can do something similar to what I want. But I haven't figured out how to do this with Insights.

Does anyone know how to count the number of occurrences of each of the IP?

amazon-cloudwatch amazon-web-services
4 Answers

The query below makes the grouping you need. I saw it here.

fields @timestamp, @message
  | parse @message /(?<@ip>^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})[.]*)/
  | stats count() as requestCount by @ip
  | filter strlen(@ip) > 0
  | sort requestCount desc
user_55043120

If your goal is to analyze actual remote IP addresses and you're using an ELB, then it's far better to capture the ELB access logs and analyze those. This will provide the following benefits:

  • Access to source request data, including client IP and SSL parameters.
  • Timing information, including timing for failed requests.
  • No need to write a custom log format that exposes request headers.

AWS Athena has pre-built queries for accessing those logs, and Athena's SQL-like query language gives you much more flexibility than CloudWatch Logs Insights. You can also ship the logs to CloudWatch and use Insights (which recognizes the ELB log formats), but this seems like unnecessary work.

user_114427350

Identify the IP wherever it is in the log:

fields @timestamp, @message
  | parse @message /(?<@ip>^*([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})[.]*)/
  | stats count() as requestCount by @ip
  | sort requestCount desc
user_7102840

"Elastic Load Balancing logs requests on a best-effort basis. We recommend that you use access logs to understand the nature of the requests, not as a complete accounting of all requests. "

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html

It is worth bearing in mind if accuracy is of high importance and using the ELB logs. I have done some work in the past analysing HTTP access logs with ELB logs and found a fairly substantial difference in the request counts between them (http logs being accurate)

user_82320310
Related