Elasticsearch regex query in java not working as intended

Viewed 459

I have a Java app that retrieves logs stored in an ElasticSearch. The logs are stored like this (this is what you retrieve from ElasticSearch):

{
 "took":1013,
 "timed_out":false,
 "_shards":{"total":40,"successful":40,"failed":0},
 "hits":{"total":28,"max_score":null,"hits":
    [
      {
      "_shard":"[logstash-2017.09.06][0]",
      "_node":"_G934CTGTjKypnI_D1b1Lg",
      "_index":"logstash-2017.09.06",
      "_type":"logs",
      "_id":"AV5WyiTlbV8ga6rEI4b8",
      "_score":null,
      "_source":{"@timestamp":"2017-09-06T10:44:01.691Z",
      "@version":"1",
      "message":"{
        \"log\":\"2017-09-19 09:26:09,149 INFO [com.mycompany.class.MyClass] (default task-23) Some log to retrieve\",
        \"stream\":\"stderr\",
        \"docker\":{
            \"container_id\":\"61b34e11002c636b289e7c40d6fbc6718e0deec58bf8a3410d598e3bd561672d\"
            },
        \"metadata\":{
            \"container_name\":\"router\",
            \"namespace_name\":\"default\",
            \"cluster_name\":\"cluster\"
            }
         }"
      },
      "sort":[1504694641691]
      }
    ]
  }
}

To get only logs that contains, for example, the word 'INFO', I want to query the "message", but the log is inside \"log\", and I want to query only for words that are inside \"log\".

I thought maybe if I query "message.log" it could work, but it didn't. It isn't a nested json ( "message":{key:value, key:value} ), it's "message":"{string}" (there are double quotes) :_(

It would be easy if the logs where stored like "log":"The log" with nothing else, but I can't change the behaviour of the logstash that is putting the logs in the ElasticSearch.

So I tried using regex ( QueryBuilders.regexpQuery("message", "Some_regex") ) with the following regex:

.*\"log\\\":\\\".*INFO.*},\\\"metadata\\\":{

I know that this regex also affects \"stream\" or \"docker\", but it's not a problem.

I tested this regex in http://regexr.com/ and https://regex101.com/ and it should work, but when I do the query, I find 0 results (and there should be results).

I trial-error tested more regex, but it didn't find results if I added anything after

.*\"log\\\":\\\".*INFO

I'm not very used to regex, and I did want to successfully accomplish it without your help, but I'm a bit lost right now...

Thank you in advance, and sorry for my bad English. Thank you!

1 Answers
Related