I need to back up my prod server Firestore DB hourly. I know about exportDocuments but it incurs one read operation per document exported. I have more than 3 million and these are increasing day by day.
Is it possible to export docs that are added/updated in a given period like the last 1 hour?
I already have Cloud Scheduler + Cloud Pub/Sub + function-based backup system. It is backing up all the docs. It is costing too much.
If you need to schedule some operations in Firestore, you can consider using Cloud Scheduler, which allows you to schedule HTTP requests or Cloud Pub/Sub messages to Cloud Functions for Firebase that you deploy.
If you need to get the documents that are added/updated in a given period of time, like the last 1 hour, then don't forget to add a timestamp field to your documents. In this way, you can query based on that timestamp field.
To get docs that are added/updated in a given period like in the last 1 hour, add a field to the document, say lastUpdated, and keep its value current with every insert/update.
Then query for incremental documents like, where("lastUpdated", ">", "lastExportTimestamp") from the backup function, lastExportTimestamp being the time of last export (and may be stored in a separate collection).
See an example here.
Hope this clarifies, else leave a comment.
P.S. Please be advised that this approach may still need a full periodic backup (say daily), for ease of restore process, if/when required.
