Fluentd unable to connect to AWS elasticsearch service using fluent-plugin-elasticsearch

user_3040389 0
Viewed 2133

We are using AWS Elasticsearch Service and fleuntd to push the logs of microservice. We have installed fluentd on EC2 instance using docker based configuration. We have followed the steps mentioned in the https://docs.fluentd.org/container-deployment/docker-compose and also it was working pretty fine till last week. There is no change in the configuration of both elastcicsearch service and ec2 instance we are provisioning. Below is the error logs we are getting.

2021-08-05 20:35:57 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method host_unreachable_exceptions' for #<Elasticsearch::Transport::Client:0x00007fe5231b5e08>" 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/elasticsearch_index_template.rb:41:in rescue in retry_operate' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/elasticsearch_index_template.rb:39:in retry_operate' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:487:in handle_last_seen_es_major_version' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:339:in configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:178:in configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:132:in add_match' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in block in configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in each' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:146:in configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in run_configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:714:in block in run_worker' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:966:in main_process' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:706:in run_worker' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:364:in <top (required)>' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/bin/fluentd:15:in <top (required)>' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/bin/fluentd:23:in load' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/bin/fluentd:23:in '

Below is the configuration used.

docker-compose.yml

version: '3'
services:
  fluentd:
    build: ./fluentd
    volumes:
      - ./fluentd/conf:/fluentd/etc
    ports:
      - "8880:8880"
      - "8880:8880/udp"

Dockerfile

FROM fluent/fluentd:v1.13.3
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.5"]
USER fluent

fluentd.conf

<source>
  @type forward
  port 8880
  bind 0.0.0.0
</source>

<match **>
    host ES_HOT
    port 443
    scheme https
    logstash_format true
    logstash_prefix "fluentd-logs"
    logstash_dateformat "%Y%m%d"
    user "elastic"
    password xxxxxx
    flush_interval 10s
    fail_on_detecting_es_version_retry_exceed false
    verify_es_version_at_startup false
    max_retry_get_es_version 2
    default_elasticsearch_version 7.10
    request_timeout 60s
    reload_on_failure true
</match>

Our configuration with AWS Elasticsearch Service 7.10, Fluentd v1.12.0-debian-1.0 and fluent-plugin-elasticsearch 4.3.3, was working perfectly fine till last week. One of the ec2 instance which was provisioned 10 days back, was able to connect and push the logs as well. But when we restarted that instance its also stopped working with the same error. We have Terraform scripts that provision all infrastructure. Later, we have tried different versions combinations as well. But nothing worked out.

Thanks in advance

amazon-ami aws-elasticsearch fluentd docker amazon-ec2
1 Answers

I had a similar issue. Digging into the matter, I found that the fluent-plugin-elasticsearch plugin doesn't version-lock its dependencies, including elasticsearch-transport and elasticsearch-api, and that a recent update somewhere in one of those led to some kind of incompatibility.

The fix that worked for me was to explicitly install the dependencies at the latest issue-free version to avoid having the fluent-plugin-elasticsearch install pull the latest issue-causing version.

Proposed Modification to your Dockerfile

FROM fluent/fluentd:v1.13.3
USER root
RUN ["gem", "install", "elasticsearch", "--no-document", "--version", "7.13.3"]
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.5"]
USER fluent
user_66811500
Related