How to iterate firestore batch writes to perform more than 500 operations in Flutter?

user_15395408 0
Viewed 571

This isn't a duplicate because I did not find any question or solution for writing more than 500 documents using batches in FLUTTER. I've seen answers for this in other frameworks or languages but I can't understand how to implement this in flutter. I need to update documents in my collections using batch write but a batched write can only contain 500 operations. So how do I write more than 500 documents to firestore? I've seen iterating as one of the solution but how do I implement that in flutter?

This is how I am performing single batch write:

Future<void> batchUpdate() {
  WriteBatch batch = FirebaseFirestore.instance.batch();

  return users
      .get()
      .then((snapshot) {
        for (DocumentSnapshot document in snapshot.docs) {
          document.reference.update(
            {
              'totalScore': 0,
            },
          );
        }
        return batch.commit();
      })
      .then((value) => ScaffoldMessenger.of(context).showSnackBar(snackBar))
      .catchError(
        (error) => ScaffoldMessenger.of(context).showSnackBar(
          SnackBar(
            content: Text(error),
          ),
        ),
      );
}
google-cloud-firestore firebase flutter
2 Answers

A batched write has a limit of 500 operations, a hard limit. You can simply perform multiple batches, to add more documents, but you can't exceed the limit.

One thing you can do is, by creating a counter variable and incrementing its value every time an operation is started to the batch. Then create an if statement and every time you increment the counter, check to see if it reached 500. At that time, commit the current batch, reset the counter and start a new batch, picking up where you left off. Do this till you finish all the batch writes. Or we can create a BatchedWrite object by calling batch(), fill that until it has a maximum capacity of 500 documents, and then write it to Firestore.

Writing 1,000 documents takes about 2.8 seconds with this approach, so the throughput is roughly 357 document writes per second.

You can try to implement the above logic in Flutter code as I am not much familiar with Flutter. But anyone who is looking for a JS code for the above logic can see below code references:

async function testBatchedWrites(datas) { 
let batch = admin.firestore().batch(); 
let count = 0; 
while (datas.length) {
 batch.set(collection.doc(Math.random().toString(36).substring(2, 15)), datas.shift()); 
if (++count >= 500 || !datas.length) { 
await batch.commit(); 
batch = admin.firestore().batch(); 
count = 0;
 } } }

Or this,

// prepare the batch 
let currentBatch = firebase.firestore().batch(); 
let currentBatchSize = 0; 
const batches = [ currentBatch ]; // add each doc's deletion to the batch 
docs.forEach((doc) => { 
// when batch is too large, start a new one 
if (++currentBatchSize >= 500) { 
currentBatch = firebase.firestore.batch(); 
batches.push(currentBatch); 
currentBatchSize = 1; } // add operation to batch 
currentBatch.delete(doc.ref); }) // commit the changes 
await Promise.all(batches.map(batch => batch.commit()));
user_158033650

I think this is a very good question that deserves more attention.

One way to solve this is to group the documents in batches of 500 (or less) and assign a batch to each group.

Note: this code leverages group from core-js since it is not part of ES lib as of today

import { DocumentData, DocumentSnapshot, Firestore, WriteBatch } from "@google-cloud/firestore"
import group from 'core-js-pure/actual/array/group'
export const groupBy = group


export interface IndexedDoc<T> {
  index: number,
  doc: T
}

export declare type BatchWriteCallback = (doc: DocumentSnapshot<DocumentData>, index: number, batch: WriteBatch) => Promise<void>


export const batchWrites = async (db: Firestore, documents: DocumentSnapshot<DocumentData>[], callback: BatchWriteCallback, countPerBatch = 500) => {
  if (countPerBatch > 500) {
    throw new Error('A Firestore WriteBatch cannot exceed 500 writes per batch')
  }
  const docs = documents.map((doc, index) => ({
    doc,
    index
  }) as IndexedDoc<DocumentSnapshot<DocumentData>>)

  const groups = groupBy(docs, (doc: IndexedDoc<DocumentSnapshot<DocumentData>>, index: number) => Math.floor(index/countPerBatch))
  const indexedGroups = Object.keys(groups).map((group) => ({
    batch: db.batch(),
    docs: groups[group] as IndexedDoc<DocumentSnapshot<DocumentData>>[]
  }))
  await Promise.all(indexedGroups.map(async group => {
    await Promise.all(group.docs.map(async indexedDoc => {
      await callback(indexedDoc.doc, indexedDoc.index, group.batch)
    }))
    await group.batch.commit()
  }))
}

To use the function:

   const db = getFirestore(...)
   const docs = (await db.collection(...).get()).docs // can be QueryDocumentSnapshot<DocumentData>[]

   await batchedWrites(db, docs, ((doc, index, batch) => {
      batch.set(doc.ref, { ... })
   }, 500)

Note: If you are making more than one batch write per document, then reduce the countPerBatch to compensate. For example, if you were making (at most) two writes per document, then reduce the countPerBatch to 250. This is needed to make sure that the total writes per batch stays within 500.

user_36616300
Related