Opensearch throws 429 error when Fluentbit outputs log under heavy load

user_5809833 0
Viewed 367

With the below fluentbit configuration we are getting errors from opensearch under heavy load.

Http bulk requests to opensearch by fluentbit(respresenting 429 errors as spike) enter image description here

Fluentbit config:

[INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     400M
        storage.type      filesystem
        Skip_Long_Lines   On
        Refresh_Interval  1
        Rotate_Wait       600
[OUTPUT]
        Name                es
        Match               kube.*
        Host                ${ES_HOST}
        Port                ${PORT}
        Buffer_Size         False
        AWS_Auth            Off
        AWS_Role_ARN        ${ES_ARN}
        AWS_External_ID     ${ES_IAMROLE}
        HTTP_User           ${ES_USER}
        HTTP_Passwd         ${ES_PASSWD}
        tls                 On
        tls.verify          Off
        Trace_Output        ${TRACE_OUTPUT}
        Trace_Error         On
        Replace_Dots        On
        Index               fluentbit
        Type                flb
        AWS_Region          ${AWS_REGION}
        Logstash_Format     On
        Logstash_Prefix     ${ES_LOGSTASHPREFIX}_app_log
        Logstash_DateFormat %Y.%m.%d
        Retry_Limit         10
        storage.total_limit_size  1G

For resolving this we have upgraded our opensearch instance type from r5.xlarge.search(4 nodes) to r5.2xlarge.search(3 nodes) but that also didn't solve the issue. We have also increased the ES index refresh_interval to 60s but that didn't help.

We read that output to ES from fluentbit can be controlled via buffering so we decreased Mem_Buf_Limit to 400M and it didn't help.

Can someone help if can try any other things or we are missing something.

fluent-bit opensearch performance
1 Answers

The issue here is not that of fluentbit but is of opensearch/elasticsearch.
The HTTP 429 errors (es_request_rejected_exception) in ES occur when too many requests are sent to the cluster, than what the thread pool for it can handle. The thread pool in OpenSearch for different tasks are allocated differently with search operations getting a larger share. The option to manually modify thread pool allocation is not available for versions 5.1 and later.
You can try to resolve this by few ways.
1: Refresh rate (you already did that and it didn't help).
2: Change the indexing speed. Try to send logs with an interval greater than your current.
3: Upscale (you did and it didn't work either)
You can get an idea with the following formula for thread pools.
Number of thread pools allocated for writes = Number of Virtual CPUs (your case)
Number of thread pools allocated for search = ((3 * Number of virtual CPUs)/2) + 1

So, I am guessing your issue here is a big number of shards! You can either decrease the shards for each index or if you are having this issue only once in a while when there is extra load, you can change the replica count to 0 and when the period is finished, change it back to the original.

Check these two links to find out more about optimizing your ES domain.

indexing performance
Best practices

user_92579910
Related