Understanding Firestore's recommended max client push rate of 1 document/second

user_2884613 0
Viewed 260

Context

My team is building a next-gen, realtime, virtual book club (not really, but it simplifies my app's actual use case) with Firestore.

When it's time for a book club to meet, users will join their respective book club and vote on which books they'd like to read next. Book clubs tend to have between 10-15 users, and each club tends to be voting on between 25-30 books. Each user can cast their votes on as many books as they wish. These book club voting sessions last around 3-4 minutes and involve all of the club's users casting their votes in this time period.

Current Firestore Architecture

My data structure looks like:

  • bookClubs (collection)
    • bookClub1 (document)
      • books (collection)
        • book1 (document)
        • book2 (document)
        • ...
        • bookX (document) where X is usually < 30
      • users (collection)
        • user1 (document)
        • user2 (document)
        • ...
        • userY (document) where Y is usually < 15
    • bookClub2 (document)
      • ...

Votes are stored in an array of maps on the book document, where each map contains the user's uid, name, and other metadata. Each of the book club users are listening for changes using snapshotChanges() on the bookClubX document, along with the nested books collection and the nested users collection.

We do this so that each user client can see in realtime how many votes are being cast on each book and each book's voters.

Limitations

In development, things seemed to work great with clubs that have between 3-4 users and are voting on 8-12 books. Last night, we had our first real book club meeting in production with 15 users and around 30-35 books, and things crashed and burned. Devices became unresponsive, mobile web browsers began to crash, and data got out of sync.

After adjourning the book club meeting and having to sadly vote with old school paper and pen, our dev team went back to the drawing board to understand where things could've failed.

We came across the following recommended best practice in the Firestore docs:

Keep the rate of documents the database pushes to an individual client under 1 document/second.

In our case, each user who joins a bookClubZ document and calls snapshotChanges() has to pull down the bookClubZ document along with each book document and user document in the two bookClubZ sub-collections. This amounts to 1 bookClubZ doc + 30 book docs + 15 user docs = ~46 docs that are being read on initialization for each and every book club user client, not to mention every update that occurs when a user casts a vote on a book.

Have we blown past the capabilities of Firestore? It definitely seems like we've grossly violated the recommended best practice above. Is this scenario better suited for something like Socket.io, PubNub, Ably, etc.? Is our Firestore data structure dumb and inefficient?

google-cloud-firestore firebase
1 Answers

You are doing 46 reads per second, which is very little. Even when you have massive numbers of users, these read operations will continue to scale very well.

The limit you quote is specific to write throughput to a single document. So you can (on average) write once to each document per second. It's actually more complex than that, and has to do with the indexes that Firestore will have to update, and the fact that all data is committed to multiple data centers before the operation is considered complete. But the 1 write per document per second is a good conservative estimate to keep in mind when designing your data structure.

The Firestore will not crash when you go above this limit though; your writes will merely be queued until the server can process them.

This is just a quick explanation of the limit you quote. It's impossible to say what your users experienced without seeing the minimal code that we can use to reproduce it. My first question would be whether you're on the free plan, and if so, check how many document reads happened. It's quite easy to rack up reads per user with a naïve data structure, and Firestore will stop working if you hit the quota limit of your project/plan.

user_2091030
Related