Elasticsearch: some fields appear immediately and some with delay

user_10095977 0
Viewed 279

I have a very strange situation

After sending documents to the ES index and creating documents, some fields in a document appear immediately, and some with a delay. That is, some fields have null,[]and0 (depending on the field type) instead of the values ​​sent to the index.

After 5-10 minutes, these fields are filled with the actual values.

This situation arises when I bulk a lot of documents (up to 500). When I sent the documents one by one, this did not occur.

The search for this problem in Google did not give anything.

What can be changed in the index or bulk settings so that documents are not displayed in the search with empty values?

Elasticsearch version: 7.6.0

Update:

For more context, I've added print screens:

1) This is how the document looks in the Kibana immediately after adding: enter image description here

2) And so the document looks after 5-10 minutes: enter image description here

In the index settings refresh_interval set to 1s

Adding refresh = "wait_for" to the bulk request didn't change anything

elasticsearch
1 Answers

Please check the refresh parameter of your index, default is 1 second but can be tweaked. Also please check when you are sending one by one you are not doing an explicit refresh or using wait_for param in the request.

According to official doc wait_for param

Wait for the changes made by the request to be made visible by a refresh before replying. This doesn’t force an immediate refresh, rather, it waits for a refresh to happen. Elasticsearch automatically refreshes shards that have changed every

Solution: For your bulk request, use wait_for param, using this will not return the result until refresh happens and once you get the response, values which you are sending will be available for search.

Further read https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-refresh.html#_choosing_which_setting_to_use

user_40394310
Related