Firebase Firestore emulator slow for bulk import

Viewed 331

The local Firestore emulator for firebase seems fairly slow when dealing with bulk data.

I'm trying to simulate a production batch import of ~5,000,000 documents, but after a few thousand records the server is so overwhelmed it slows to a rate of around 5 records/second.

On the topic of Transactions and Batched Writes firebase says...

Note: For bulk data entry, use a server client library with parallelized individual writes. Batched writes perform better than serialized writes but not better than parallel writes. You should use a server client library for bulk data operations and not a mobile/web SDK.

That implies to me that I should simply not use the mobile/web SDK, and should make sure I use a server-side SDK/API for imports.

I'm already using the server client library for node.js.

  const batch = firestore.batch();

  for (const record of records) {
    batch.set(
      firestore.collection(collection).doc(record.key),
      record,
    );
  }
  await batch.commit();

...and this issue makes me think the local emulator (or firestore in general) can't keep up with a high load.

0 Answers
Related