Firebase Functions and Express: listen to firestore data live

Question

I have a website that runs its frontend of Firebase Hosting and its server which is written using node.js and Express on Firebase Functions

What I want to have redirect links from my website so I can map for example mywebsite.com/youtube to my youtube channel. the way I am creating these links is from my admin panel, and adding them to my Firestore database.

My data is roughly something like this:

The first way I approached this, is by querying my Firestore database on every request, but that is heavily expensive and slow.

Another way I tried to approach this is by setting some kind of background listener to the Firestore database which will always provide up to date data. but unfortunately that did not work because Firebase Functions suspends the main function when the current request execution ends.

lastly, which is the most convenience way, I configured an api route, which will be called from my Admin Panel when any change happens to the data, and I would save the new data to some json file. I tried this on my local but it did not work on production because appearently Firebase Functions is a Read-only system, so we can't edit any files after they are deployed. after some research I found out that Firebase Functions allows writing to the tmp directory, so I went forward with this, and tried deploying it. but again, Firebase Functions was resetting the tmp folder when some request execution ends.

here is my api request code which updates the utm_data.json file in the tmp directory:

// my firestore provider
const db = require('../db');

const fs = require('fs');
const os = require('os')

const mkdirp = require('mkdirp');

const updateUrlsAPI = (req, res) => {
    // we wanna get the utm list from firestore, and update the file 
    // tmp/utm_data.json

    // query data from firestore
    db.collection('utmLinks').get().then(async function(querySnapshot) {
        try {
            // get the path to `tmp` folder depending on
            // the os running this program
            let tmpFolderName = os.tmpdir()

            // create `tmp` directory if not exists
            await mkdirp(tmpFolderName)

            let docsData = querySnapshot.docs.map(doc => doc.data())

            let tmpFilePath = tmpFolderName + '/utm_data.json'

            let strData = JSON.stringify(docsData)
            fs.writeFileSync(tmpFilePath, strData)

            res.send('200')
        } catch (error) {
            console.log("error while updating utm_data.json: ", error)
            res.send(error)
        }
    });
}

and this is my code for reading the utm_data.json file on an incoming request:

const readUrlsFromJson = (req, res) => {

    var url = req.path.split('/');
    // the url will be in the format of: 'mywebsite.com/routeName'
    var routeName = url[1];

    try {
        // read the file ../tmp/utm_data.json
        // {
        //   'createdAt': Date
        //   'creatorEmail': string
        //   'name': string
        //   'url': string
        // }
        // our [routeName] should match [name] of the doc
        let tmpFolderName = os.tmpdir()

        let tmpFilePath = tmpFolderName + '/utm_data.json'

        // read links list file and assign it to the `utms` variable
        let utms = require(tmpFilePath)

        if (!utms || !utms.length) {
            return undefined;
        }

        // find the link matching the routeName
        let utm = utms.find(utm => utm.name == routeName)

        if (!utm) {
            return undefined;
        }

        // if we found the doc,
        // then we'll redirect to the url
        res.redirect(utm.url)
    } catch (error) {
        console.error(error)
        return undefined;
    }
}

Is there something I am doing wrong, and if not, what is an optimal solution for this case?

user_13130697 · Answer

You can initialize the Firestore listener in global scope. From the documentation,

The global scope in the function file, which is expected to contain the function definition, is executed on every cold start, but not if the instance has already been initialized.

This should keep the listener active even after the function's execution has completed until that specific instance is running (which should be about ~30 minutes). Try refactoring the code as shown below:

import * as functions from "firebase-functions";
import * as admin from "firebase-admin";

admin.initializeApp();

let listener = false;

// Store all utmLinks in global scope
let utmLinks: any[] = [];

const initListeners = () => {
  functions.logger.info("Initializing listeners");
  admin
    .firestore()
    .collection("utmLinks")
    .onSnapshot((snapshot) => {
      snapshot.docChanges().forEach(async (change) => {
        functions.logger.info(change.type, "document received");
        switch (change.type) {
          case "added":
            utmLinks.push({ id: change.doc.id, ...change.doc.data() });
            break;

          case "modified":
            const index = utmLinks.findIndex(
              (link) => link.id === change.doc.id
            );
            utmLinks[index] = { id: change.doc.id, ...change.doc.data() };
            break;

          case "removed":
            utmLinks = utmLinks.filter((link) => link.id !== change.doc.id);

          default:
            break;
        }
      });
    });
  return;
};


// The HTTPs function
export const helloWorld = functions.https.onRequest(
  async (request, response) => {
    if (!listener) {
      // Cold start, no listener active
      initListeners();
      listener = true;
    } else {
      functions.logger.info("Listeners already initialized");
    }

    response.send(JSON.stringify(utmLinks, null, 2));
  }
);

This example stores all UTM links in an array in global scope which won't be persisted in new instances but you won't have to query each link for every request. The onSnapshot() listener will keep utmLinks updated.

The output in logs should be:

If you want to persist this data permanently and prevent querying in every cold start, then you can try using Google Cloud Compute that keeps running unlike Cloud functions that timeout eventually.

Firebase Functions and Express: listen to firestore data live

1 Answers