How to parse log with json using Filebeat to store in Elasticsearch

Viewed 190

The log file that I am using has logs in the following format:

2021-07-29 14:51:00,668 INFO audit [http-nio-8080-exec-4] {"user": "John", "country": "USA"}
2021-07-29 14:51:00,668 INFO audit [http-nio-8080-exec-4] {"user": "Jack", "country": "CA"}
2021-07-29 14:51:00,668 INFO audit [http-nio-8080-exec-4] {"user": "Josh", "country": "USA"}

I want my document to look like the following (excluding the elasticsearch metadata fields) in Elasticsearch

{
  "user": "John",
  "country": "USA"
}

How can I store the data in above format on Elasticsearch using FileBeat?

Option 1:

I have tried the following options in Filebeat. But it throws error as "error in decoding..."

  json.keys_under_root: true
  json.add_error_key: true

Option 2:

I haven't disabled the above json options in Filebeat. But log entry stored as single value in message key as below

{
"message": "2021-07-29 14:51:00,668 INFO audit [http-nio-8080-exec-4] {"user": "John", "country": "USA"}"
}
0 Answers
Related