How to reliably determine when an Azure Cognitive Search index is up to date?

user_4407112 0
Viewed 164

Azure Cognitive Search is eventually consistent - writes to the service return successfully but the writes are not materialized in the search index for a short period of time.

We are using Azure Cognitive Search in an eventually consistent event sourced CQRS architecture, where an Azure Search index is used as a projection of the event stream. We use websockets to notify connected clients when a projection has been updated, so that they can re-query it to fetch the latest data.

This presents a challenge with Azure Search, because when we notify a client that the index has been updated, the client may query the index before it can provide the most up to date data.

Does Azure Cognitive Search provide any built in ability to determine when a given write will be queryable? If not, what patterns can be used to achieve what we want?

azure-cognitive-search azure
1 Answers

I am not aware of any functionality in Azure Search that allows you to submit content with a callback that confirms that the content is indexed. However, I have used search engines before where this was an option. When submitting content you could choose between different wait options:

  • Fire and forget (quickest, return immediately)
  • Wait for confirmed storage (reasonably quick)
  • Confirmed indexed (potentially very slow depending on overall indexing load)

It seems like you want the last option. You could create a function that submits a batch of content and then queries until that content is available. For this to work, you would need an indicator on each record that you can use to confirm via a query that the new records are in fact indexed. I always include a timestamp property on all my records and it would do the job in this case.

Use case: You have an index with 500 items. You then have a batch of 10 updated items where 5 are updates and 5 are new records. You add a timestamp to all of these records, submit the batch of 10 and then query in a loop. Once the query confirms that you have 10 or more records with a timestamp higher or equal to the time you submitted your batch, you know the index is updated.

user_1348500
Related