iOS firestore addSnapshotListeners across multiple clients stop receiving updates for 20-30 seconds

We once daily create a Firestore document (~4kB) which gets updated in regular intervals (every 10 seconds) in a timeframe of 15 minutes. Our iOS clients subscribe to changes on the document using addSnapshotListener and subsequently write changes to a subcollection of the document.

This setup has been running smoothly for about a year now. Since we have experienced a significant increase in new clients (from 300 to ~1.5k), once during the 15 minutes, most of the clients won't receive snapshot updates for about 20-30 seconds. Before and after this occurrence snapshots are received as expected.

So far we haven't been able to reproduce this issue using only a handful of clients. We're still working on implementing a stress test to try to reproduce the observed behavior.

Could this in some form be caused by automatic scaling to accommodate the number of clients listening for snapshots?

Any suggestions would be greatly appreciated.

Versions

• Xcode version: 13.0 (13A233)
• Firebase SDK version: 7.11.0

Listener Initialization

init(subscriber: S, ref: DocumentReference) {
    self.subscriber = subscriber
    self.listener = ref.addSnapshotListener(includeMetadataChanges: true) { [weak self] snapshot, error in
        self?.receive(snapshot: snapshot, error: error)
    }
}

Update

After further testing we don't think the issue is caused by unexpected behavior of our client code or the firebase iOS SDK. Rather it seems that what our clients are experiencing is caused by write operation spikes exceeding the recommended limit of 1,000 writes per second on a subcollection of the document the snapshot listeners are attached to.

The behavior of snapshot listeners not receiving updates for ~25s does not always occur and so far we still have not been able to reproduce the behavior in testing using the same document structure and 2,000 writes per second.

The guidlines state:

Keep the rate of write operations for an individual collection under 1,000 operations/second.

Our plan to fix the issue is manually sharding the collection, so write operations will never exceed the limit for a single collection. This solution should be viable until we reach 10,000 clients and therefore reach the limit of 10,000 write operations per second for the database.

Incorrect handling of the 500/50/5 rule is also something we're still monitoring.