UnknownException in Logstash HTTP output plugin

user_398230 0
Viewed 748

Some requests in my logstash failed the http output plugin, and the log showing

[2020-10-16T18:44:54,574][ERROR][logstash.outputs.http ] [HTTP Output Failure] Could not fetch URL {:url=>"https://www.example.com/api.php", :method=>:post, :body=>"{"id":"test"}", :headers=>{"Content-Type"=>"text/plain"}, :message=>"Connection reset", :class=>"Manticore::UnknownException", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in block in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:239:in send_event'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:175:in send_events'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:124:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:239:in block in start_workers'"], :will_retry=>false}

What would be the reason? And how to fix it, i.e. retry again later?

My logstash output configuration is

output {
    if [message] == "..." {
        http {
            format=>"message"
                http_method=>"post"
                url=>"xxx"
                message=>'...'
        }
    }
}
logstash
1 Answers

The Logstash Http output plugin uses Manticore HTTP Client. It will do automatic retries if the exception falls in below categories. Check this.

  ::Manticore::Timeout,
  ::Manticore::SocketException,
  ::Manticore::ClientProtocolException,
  ::Manticore::ResolutionFailure,
  ::Manticore::SocketTimeout

For your use-case the http client is getting Manticore::UnknownException which is not part of the default list for retries. And I do not see a way to override it using configuration.

I can think of two options.

  1. Debug why the connection reset is happening at network level and resolve the underlying issue. Ex: firewall or network whitelist at the API end, API requires mutual auth
  2. If 1 is not an option and you are sure that retrying is an option and will solve your problem, fork the github repo, add Manticore::UnknownException to the list of retry-able exceptions and use that version of plugin.
user_143660770
Related