How to import test data into Firebase Firestore emulator from production for easy testing?

user_577878 0
Viewed 3244

I am building a firebase function that listens to a trigger and sends a push notification to users. The trigger is based on firestore data that's too complex for me to manually re-recreate every time on firestore emulator.

I tried emulating functions, but hooking up to firestore production, but seems like in this case, trigger functions don't work

What I want to do is export my data from my production Firestore then import it into my emulator firestore, so that at least, the copy I am working with is closely mimicking what I have on prod; to be clear, the data is small, so I am not worried about downloading terabytes of data.

I found a way of importing data into an emulator, but not sure if this would work with production data or how I would dump the data from production.

firebase emulators:start --import=./some-directory
firebase-tools google-cloud-firestore firebase node.js
3 Answers

You can manually export from Production and then import into the Firestore emulator.

Start by doing an export of your production data to a file. In the example, I export just the "users" collection within an onRequest function so I can kick-off by calling the URL. Note, this this function will not handle subcollections.

exports.exportFirestore = functions.https.onRequest(async (req, res) => {
  const fs = require("fs");

  const collection = "users";
  const fileName = "fs-export.json";
  const exportedData = {};
  exportedData[collection] = {};

  await admin
    .firestore()
    .collection(collection)
    .get()
    .then((snapshot) => {
      return snapshot.forEach((doc) => {
        exportedData[collection][doc.id] = doc.data();
      });
    })
    .catch(console.error);

  fs.writeFile(fileName, JSON.stringify(exportedData), (err) => {
    if (err) {
      console.log(err);
    } else {
      console.log("Firestore Export Complete.");
      res.send("Firestore Export Complete.");
    }
  });
});

Next, import the file into the local Firestore emulator. Very important: be sure to have the Firestore emulator running otherwise you'll be importing your data back to prod. Be sure to check that Firestore is running by checking the web UI: http://localhost:4000. Start the emulator with: firebase emulators:start

exports.importFirestore = functions.https.onRequest(async (req, res) => {
  const fs = require("fs");

  let collection;
  const fileName = "fs-export.json";
  const exportedData = {};
  exportedData[collection] = {};

  fs.readFile(fileName, "utf8", async (err, data) => {
    if (err) {
      return console.log(err);
    }

    const arr = JSON.parse(data);

    const batch = admin.firestore().batch();

    for (let i in arr) {
      collection = i;
      for (let doc in arr[i]) {
        if (arr[i].hasOwnProperty(doc)) {
          const ref = admin.firestore().collection(collection).doc(doc);
          batch.set(ref, arr[i][doc]);
        } else {
          console.log("Missing:", JSON.stringify(doc, null, 2));
        }
      }
    }

    await batch
      .commit()
      .then(() => {
        return console.log("Import to Firestore Complete");
      })
      .catch(console.error);

    return res.send("Import to Firestore Complete");
  });
});
user_77119680

My answer is heavily inspired by @Geoffrey Bourne's answer, but I had to modify some stuff and figure out more details to get it working.

First, I upload the exportFirestore to Cloud Functions (production). When I run it through this URL https://us-central1-<project-id>.cloudfunctions.net/exportFirestore, I get a file DOWNLOADED, as Cloud Functions are read-only

The below code is for one collection named fl_content, I'll consider expanding it to multiple collections

export const exportFirestore = functions.https.onRequest(async (req, res) => {
    const collection = "fl_content";
    const exportedData: any = {};
    exportedData[collection] = {};

    await admin
        .firestore()
        .collection(collection)
        .get()
        .then((snapshot) => snapshot.forEach((doc) => exportedData[collection][doc.id] = doc.data()))
        .catch(console.error);

    const data = JSON.stringify(exportedData);
    res.setHeader('Content-disposition', 'attachment; filename=fire-export.json');
    res.setHeader('Content-type', 'application/json');
    res.write(data, function () {
        res.end();
    });
})

Once you have the file fire-export.json downloaded, put it inside the functions folder. Then open the URL for the import function (locally) http://localhost:5001/<project-id>/us-central1/importFirestore. Make sure the collection variable is the same in the export and import.

export const importFirestore = functions.https.onRequest(async (req, res) => {
    const fs = require("fs");
    const collection = "fl_content";
    const fileName = "fire-export.json";
    const exportedData: any = {};
    exportedData[collection] = {};

    fs.readFile(fileName, "utf8", async (err: any, data: any) => {
        if (err) {
            res.send(err);
            functions.logger.error(err)
            return;
        }
        const arr = JSON.parse(data);
        const batch = admin.firestore().batch();

        for (const i in arr) {
            for (const doc in arr[i]) {
                if (arr[i].hasOwnProperty(doc)) {
                    const ref = admin.firestore().collection(collection).doc(doc);
                    batch.set(ref, arr[i][doc]);
                } else {
                    functions.logger.error("Missing:", JSON.stringify(doc, null, 2));
                }
            }
        }

        await batch
            .commit()
            .then(() => console.log("Import to Firestore Complete"))
            .catch(console.error);

        res.send("Import to Firestore Complete");
    });
});
user_41574310

One way to persist your data is by exporting to directory on exit and on start import the same directory

firebase emulators:start  --import=./mydir --export-on-exit mydir
user_104095670
Related