Elastic search 403 Request throttled due to too many requests /_bulk

user_5419780 0
Viewed 3089

I am trying to sync 1 million record to ES, and I am doing it using bulk API in batch of 2k. But after inserting around 25k-32k, elastic search is giving following exception.

Unable to parse response body: org.elasticsearch.ElasticsearchStatusException
ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [POST], host [**********], URI [/_bulk?timeout=1m], status line [HTTP/1.1 403 Request throttled due to too many requests]
403 Request throttled due to too many requests /_bulk]; nested: ResponseException[method [POST], host [************], URI [/_bulk?timeout=1m], status line [HTTP/1.1 403 Request throttled due to too many requests]
403 Request throttled due to too many requests /_bulk];

I am using aws elastic search. I think, I need to implement wait strategy to handle it, something like keep checking es status and call bulk insert if status all of ES okay. But not sure how to implement it? Does ES offers anything pre-build for it? Or Anything better way to handle this?

Thanks in advance.

Update: I am using AWS elastic search version 6.8

aws-elasticsearch elasticsearch amazon-web-services
2 Answers

Thanks @dravit for including my previous SO answer in the comment, after following the comments it seems OP wants to improve the performance of bulk indexing and want exponential backoff, which i don't think Elasticsearch provides out of the box.

I see that you are putting a pause of 1 second after every second which will not work in all the cases, and if you have large number of batches and documents to be indexed, for sure it will take a lot of time. There are few more suggestions from my side to improve the performance.

  1. Follow my tips to improve the reindex speed in Elasticsearch and see what all things listed here is applicable and doing them improves speed by what factor.
  2. Find a batching strategy which best suits to your environment, I am not sure but this article from @spinscale who is the developer of java high level rest client might help or you can ask a question on https://discuss.elastic.co/, I remembered he shared a very good batching strategy in one of his webinar but couldn't find the link of it.
  3. Notice various ES metrics apart from bulk threadpool and queue size, and see if your ES still has capacity can you increase the queue size and increase the rate by which you can send requests to ES.
user_40394310

Check the error handling guide here

If you receive persistent 403 Request throttled due to too many requests or 429 Too Many Requests errors, consider scaling vertically. Amazon Elasticsearch Service throttles requests if the payload would cause memory usage to exceed the maximum size of the Java heap.

Scale your application vertically or increase delay between requests.

user_75672040
Related