Rails 5 logstash produces too many log items instead of one

user_1252574 0
Viewed 177

Is there any way I can customize the way standard errors are being logged? Starting with Rails 5, logstash receives one fatal error as a lot of error items. Rails 4 only created one item (The way I liked it)

The problem seems to be, that since Rails 5, the log_error method of the DebugExceptions middleware prints all lines of the backtrace calling logger.fatal separately, instead of Rails 4, which joins all lines to one string before calling logger.fatal only once.

Compared:

Rails 4: debug_exceptions.rb

message = "\n#{exception.class} (#{exception.message}):\n"
message << exception.annoted_source_code.to_s if exception.respond_to?(:annoted_source_code)
message << "  " << trace.join("\n  ")
logger.fatal("#{message}\n\n")

Rails 5: debug_exceptions.rb

logger.fatal "  "
logger.fatal "#{exception.class} (#{exception.message}):"
log_array logger, exception.annoted_source_code if exception.respond_to?(:annoted_source_code)
logger.fatal "  "
log_array logger, trace

I'm using logstash-logger with the :json_lines formatter over UDP.

The logstash configuration I use is working perfectly for rails 4 and other applications, but not for rails 5 (as it separates one error into several).

I know, I could use rescue_from to rescue errors in controllers, but this would not catch errors on routing or others, I guess?

Thank you in advance.

logstash kibana logging ruby-on-rails
1 Answers

Actually for error handling it's more convenient to use a dedicated exception tracking solution like rollbar, airbrake, sentry, errbit or similar. Point is in that for exceptions you need:

  1. a quick way to pinpoint location and conditions where it happens, including request parameters, user id, headers etc. by default not all these are written to logs. But most exception catchers also record additional info about the request.
  2. alerting, but not too noisy, because when something goes wrong in production - you'll quickly have hundreds of logged errors and you cannot simply forward all these to your email or im. So aggregation must be supported
  3. optional, but useful - tracking open/resolved errors

It's hard to navigate raw logs for exceptions, even if they're tagged and indexed, also there may be situation when you have a noisy non-mission-critical exception that floods the logs, and makes you miss some rare really important one.

As for catching the exception in rails - it's better to do in middleware, like all tools mentioned above (and rails itself) do:

class MyExceptionMidleware
  def initialize(app)
    @app = app
  end

  def call(env)
    @app.call(env)
  rescue => e
    # handle the error any way you like, for example:
    Rails.logger.fatal("oops, #{e.inspect}, backtrace: #{e.backtrace.join("\n")}")
  end
end

config.middleware.insert_after(ActionDispatch::DebugExceptions, MyExceptionMidleware)

(or insert_before, but that way the exception will not be logged, if you do not reraise it, or if it occurs in middleware itself, not recommended)

user_1770530
Related