I am using Rails 5.2 with Searchkick 4.4.2 and an Elasticsearch 7.9 node on AWS.
I am trying to reindex some data in my index, approximately 500k records, and am getting either a 403 throttle error, or a Faraday::TimeoutError (Net::ReadTimeout with #<TCPSocket:(closed)>).
I tried indexing with various methods :
- Object.reindex
- Object.reindex(async: true)
- Object.in_batches with a sleep time inside the block
- Dividing my collection into subsets and indexing each subset individually
What is a good way to reindex a large amount of data ? Are there some parameters that I can change in Searchkick/Elasticsearch to avoid timeouts or throttle errors ?
Thanks for your help !