I'm implementing Elastic Search for a blog where documents can be updated.
I need to perform hitless reindexing in Elastic Search that avoids races and keeps consistency. (By consistency, I mean if the application does a write followed by a query, the query should show the changes even during reindexing).
The best advice I've been able to find is that you use aliases to atomically switch which index the application is using and that the application writes to both the old index (via an write_alias) and the new index (via a special write_next_version alias) when writing during a reindexing operation, and reads from the old index (via a read_alias). Any races in the concurrent writes between reindex and the application are resolved by the document version numbers as long as the application writes to the old index first and the new index second. When the reindexing is done, just atomically switch the application's read and write aliases to the new index and delete the write_next_version alias.
However, there are still races and performance issues.
My application doesn't know there's a reindex occurring, reindex and the alias switching involved is a separate long-running process. I could use a HEAD command to find out if a special write_next_version alias exists and only write if it exists. However, that's an extra round trip to the ES servers. There's also still a race between the HEAD command and the reindex process described above that deletes the second write_next_version alias. I could just do both writes every time and silently handle the error to the usually non-existent write_next_version alias. I'd do this via a bulk API if my documents were small, but they are blog entries, they could be fairly large.
So should I just write twice every time and swallow the error on the second write? or should I use the HEAD API to determine whether the application needs to perform the second write for consistency? Or is there some better way of doing this?
The general outline of this strategy is shown in this article. This older article also shows how to do it but they don't use aliases which is not acceptable. There is a related issue on the Elastic Search github but they do not address the problem that two writes need to be done in order to maintain consistency. They also don't address the races or performance issues. (they closed the issue...)