I am in the middle of optimizing a search query using the rank_eval api, similar to what is being done in the following article https://www.elastic.co/blog/improving-search-relevance-with-data-driven-query-optimization using bayesian optimization.
Here we are using python and the scikit-optimize library for optimization, and the python elasticsearch library to handle sending the rank_eval requests. https://elasticsearch-py.readthedocs.io/en/v8.4.2/api.html#elasticsearch.Elasticsearch.rank_eval
The issue is that in the first couple of iterations of the optimization the time pr iteration is fairly fast (30sec), but then after some iterations it slows down to an ubearable time pr iteration (5min - 15min), and it also hurts normal search queries, making it nearly unusable. After stopping the optimization, the response time returns to normal.
I've tried to batch the data that is sent to the rank_eval api, but with no luck, it is still going slower and slower.
What is the best way to proceed?
Would it be an idea to redirect the traffic to the replica shard or would this not fix the issue?
I am using 5 different indexes corresponding to 5 different languages, each index has a primary shard and exactly 1 replica shard.