Extract timestamp from JSON and use it as the logstash event timestamp

Question

I hope somebody can help me out. Ik have the following setup:

shipper >> redis >> logstash >> elasticsearch

Logstash has an input, which extracts log data from redis. The following JSON line goes in redis:

{
  "@fields": {
    "level": "INFO",
    "mdc": {},
    "file": "ContextLoader.java",
    "class": "org.springframework.web.context.ContextLoader",
    "line_number": "273",
    "method": "initWebApplicationContext"
  },
  "@timestamp": "2013-12-18T11::29:29.115+01:00",
  "@message": "Root WebApplicationContext: initialization started",
  "@source_host": "test-host"
}

The logstash configuration I use is simple:

input {
  redis {
    data_type => "list"
    key => "logstash:test:host"
    codec => "json"
  }
}
filter{
  date {
    match => [ "@timestamp", "dd-MMM-YYYY:HH:mm:ssZ", "dd-MMM-YYYY:HH:mm:ss Z", "ISO8601" ]
  }
}
output {
  stdout{
    debug => true
  }
}

When I use the above configuration, I see that the timestamp of the logstash event is used (2013-12-18T12:53:14.169Z) and I get the following error from my debug screen:

Failed parsing date from field {:field=>"@timestamp", :value=>"2013-12-18T12:53:14.177Z", :exception=>#, :level=>:warn}

From this, I can conclude the following:

When the log entry comes in as input, the timestamp of the json log entry is overwritten with the timestamp event of logstash. Then, when the date filter tries to parse the timestamp, it fails because, there is no match.

I want to use the timestamp of the json log entry as my logstash event timestamp. How can I achieve this? I use the logstash version 1.3.1.

Extract timestamp from JSON and use it as the logstash event timestamp

1 Answers