FluentD - Parsing Heroku’s Logplex format

Question

We’re using FluentD to send data over to our ELK stack. Heroku sends over logs in a BULK format which includes multiple log entries, separated by a new line.

I was wondering if anyone had any experience with splitting incoming http requests in FluentD by newline? I saw examples of this in past versions < 1.0. There are also two Heroku+FluentD plugins, both of which no longer seem to work and are not maintained.

Can I use a parser to split the incoming message into multiple messages and emit each to FluentD, if so, how?
If not, is there a simpler way to get these bulk messages sent from Heroku into FluentD, split by new line?

The bulk log messages Heroku posts look something like this:

83 <40>1 2012-11-30T06:45:29+00:00 host app web.3 - State changed from starting to up 119 <40>1 2012-11-30T06:45:26+00:00 host app web.3 - Starting process with command bundle exec rackup config.ru -p 24405

So in our logging solution, we’re getting multiple rows per entry. We’ve tried multi line parsing, but that doesn’t seem to do the trick.

user_12747692 · Answer

You can achieve your goal by using the following gem

Install the gem and use one of the following configs

Use this one of your events are separated with a new line


  @type record_splitter
  tag splitted.log
  input_key message
  split_stratgey lines
  append_new_line true
  remove_new_line true

or use this one to split the lines with regex


  @type record_splitter
  tag splitted.log
  input_key message
  split_stratgey regex
  split_regex /\d+\s<\d+>.+/

to be able to process the log lines further, you can add another output match for the tag used (example below to send logs to elk)


...



  @type rewrite_tag_filter



  @type elasticsearch
  ...

FluentD - Parsing Heroku’s Logplex format

1 Answers