Bulk write to ElasticSearch from Firestore using Google Cloud Functions

user_10557732 0
Viewed 651

Currently I am using Firestore for my database and I have a users collection. Whenever a user document is created or updated to the users collection, a cloud function takes the user document and saves it in Elastisearch.

I am starting to be concerned about the scalability to this architecture. For example, suppose that several thousand cloud functions started writing documents to Elasticsearch at once, is Elasticsearch going to handle this load. Is there a better solution to this in Google cloud?

For example, can those cloud functions write the user documents in a queue and have cloud functions at the other end of the queue take a 100 documents and bulk write them to Elasticsearch.

I am new to Google cloud and would appreciate if you give me ideas, videos, and things to read.

Thanks

google-cloud-firestore elasticsearch firebase google-cloud-platform
1 Answers

ElasticSearch has no limits on number of documents it can have per index but there are some limits such as maximum doc size and bulk writes mentioned in their documentation.

Maximum Document Size: 100KB [configurable in 7.7+]

Maximum Indexing Payload Size: 10MB

Bulk Indexing Maximum: 100 documents per batch

As far as I know, Google Cloud has no full text search API.

Talking of bulk writes, if realtime availability (data to be available immediately after adding) if not a concern, then you can store the new documents in Firestore along with a timestamp they were added and a boolean value if a document has been indexes in Elasticsearch.

Then instead of running a cloud function with onCreate trigger, you can run a scheduled cloud function every N minutes which will:

  1. Query documents which have not been added in Elasticsearch
  2. Make batches of 100 (for the 1000/batch limit)
  3. Upload them to Elasticsearch

This way you are are more documents per cloud function run so that'll be a bit efficient but if you need your new data to be available immediately then this won't work.

user_131306970
Related