Elasticsearch - find IPs from which only anonymous requests came

user_13890931 0
Viewed 40

I have network logs in my Elasticsearch. Each log has an username and an IP field. Something like this:

{"username":"user1", "ip": "1.2.3.4"}
{"username":"anonymous", "ip": "1.2.3.4"}
{"username":"anonymous", "ip": "2.3.4.5"}
{"username":"user2", "ip": "3.4.5.6"}

I have a seemingly simple task: list all IP-s from which only anonymous requests came. The problem is, I can not simply filter for anonymous, because then I'll list false IP-s which appear with anonymous, but not exclusively. Manually I can do this with a 3 step process:

  1. List all unique IP-s
  2. List unique IP-s that appear with something other than anonymous
  3. Exclude items of 2nd list from the first.

But is there a way to do this with a single ES query? My first instinct was to use bool query. My current approach is this:

GET /sample1/_search
{
  "query": {
    "bool": {
      "must": {
        "wildcard": {
          "ip": "*"
        }
      },
        "must_not": {
          "term": {
            "username": "-anonymous"
          }
        }
      }
  },
  "size": 0,
  "aggs": {
    "ips": {
      "terms": {
        "field": "ip.keyword"
      }
    }
  }
}

I expect "2.3.4.5", but it returns all 3 unique IPs. I searched the web and tried different query types for hours. Any ideas?

elasticsearch
1 Answers

Please find the below mapping, sample docs, the respective query for your scenario and the response:

Mapping:

PUT my_ip_index
{
  "mappings": {
    "properties": {
      "user":{
        "type": "keyword"
      },
      "ip":{
        "type": "ip"
      }
    }
  }
}

Documents:

POST my_ip_index/_doc/1
{
  "user": "user1",
  "ip": "1.2.3.4"
}

POST my_ip_index/_doc/2
{
  "user": "anonymous",
  "ip": "1.2.3.4"
}

POST my_ip_index/_doc/3
{
  "user": "anonymous",
  "ip": "2.3.4.5"
}

POST my_ip_index/_doc/4
{
  "user": "user2",
  "ip": "3.4.5.6"
}

Aggregation Query:

POST my_ip_index/_search
{
  "size": 0,
  "aggs": {
    "my_valid_ips": {
      "terms": {
        "field": "ip",
        "size": 10
      },
      "aggs": {
        "valid_users": {
          "terms": {
            "field": "user",
            "size": 10,
            "include": "anonymous"
          }
        },
        "min_bucket_selector": {
          "bucket_selector": {
            "buckets_path": {
              "valid_users_count": "valid_users._bucket_count",
              "my_valid_ips_count": "_count"
            },
            "script": {
              "source": "params.valid_users_count == 1 && params.my_valid_ips_count == 1"
            }
          }
        }
      }
    }
  }
}

Note how I've made use of Terms Aggregation and Bucket Selector Aggregation in the above query.

I've added include part in Terms Agg so as to consider only anonymous users and the logic inside bucket aggregation is to filter out only if it is a single doc count in the top level terms aggregation for e.g. 2.3.4.5 followed by single bucket count in the second level terms aggregation.

Response:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "my_valid_ips" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "2.3.4.5",                 <---- Expected IP/Answer
          "doc_count" : 1,
          "valid_users" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "anonymous",
                "doc_count" : 1
              }
            ]
          }
        }
      ]
    }
  }
}

Hope it helps!

user_38383280
Related