Failing to implement pagination in Firebase realtime database

Question

Below is the code that I'm using to implement pagination for data retrieved from the firebase realtime database. Basically, I'm trying to get the first n content according to page number, and then getting the last n content from the data retrieved in the first query.

function getUserSnapshotOrVerifyUserId(username, idToken, cb) {
    if (username == null || username.length == 0 || idToken == null || idToken.length == 0)
        return cb({
            status: "error",
            errorMessage: "Missing params."
        }, null);
    
    admin.auth().verifyIdToken(idToken).then(decodedToken => {
        let uid = decodedToken.uid;

        admin.database().ref().child("users").orderByChild("username").equalTo(username).once('value', snapshot => {
            if (!snapshot.exists())
                return cb({
                    status: "error",
                    message: "invalid-profile"
                });

            snapshot.forEach(child => {
                const id = child.val().id;
                if (id !== uid)
                    return cb({
                        status: "error",
                        message: "Invalid ID"
                    });

                admin.database().ref("users/" + id).once("value", snapshot => {
                    if (!snapshot.exists())
                        return cb({
                            status: "error",
                            errorMessage: "user not found."
                        });
                    
                    return cb(null, id, snapshot);
                });
            });
        });
    }).catch(err => cb({
        status: "error",
        message: err
    }));
}

exports.getUserContentTestPagination = functions.https.onRequest((req, res) => {
    corsHandler(req, res, async () => {
        try {
            const username = req.body.username || req.query.username;
            const idToken = req.body.idToken;

            const limit = 2;
            const page = req.body.page || 1;

            const limitToFirst = page * limit;
            const limitToLast = limit;


            getUserSnapshotOrVerifyUserId(username, idToken, async (err, id) => {
                if(err) return res.json(err);

                const uploadsRef = admin.database().ref('uploads').orderByChild('createdBy').equalTo(id)

                const firstnquery = uploadsRef.limitToFirst(limitToFirst);
                const lastnquery = firstnquery.limitToLast(limitToLast);

                lastnquery.once("value", snapshot => {
                    res.json({
                        snapshot
                    })
                })
                
            })
        } catch (err) {
            res.json({
                status: "error",
                message: err
            })
        }
    });
});

This is returning a function timeout, however, when I try to get the first n data using firstnquery, it is returning the first n data as expected. So the problem is with lastnquery. Any help would be appreciated.

UPDATE 1:

exports.getUserContentTestPagination = functions.https.onRequest((req, res) => {
    corsHandler(req, res, async () => {
        try {
            const username = req.body.username || req.query.username;
            const idToken = req.body.idToken;

            const limit = 2;
            const page = req.body.page || 1;
            
            let lastKnownKeyValue = null;

            getUserSnapshotOrVerifyUserId(username, idToken, async (err, id) => {
                if(err) return res.json(err);

                const uploadsRef = admin.database().ref('uploads');
                const pageQuery = uploadsRef.orderByChild('createdBy').equalTo(id).limitToFirst(limit);
                
                pageQuery.once('value', snapshot => {
                    snapshot.forEach(childSnapshot => {
                        lastKnownKeyValue = childSnapshot.key;
                    });

                    if(page === 1){
                        res.json({
                            childSnapshot
                        })
                    } else {
                        const nextQuery = uploadsRef.orderByChild('createdBy').equalTo(id).startAt(lastKnownKeyValue).limitToFirst(limit);

                        nextQuery.once("value", nextSnapshot => {
                            nextSnapshot.forEach(nextChildSnapshot => {
                                res.json({
                                    nextChildSnapshot
                                })
                            })
                        })
                    }

                });

            })
        } catch (err) {
            res.json({
                status: "error",
                message: err
            })
        }
    });
});

user_209103 · Answer

It is incredibly uncommon to use both limitToFirst and limitToLast in a query. In fact, I'm surprised that this doesn't raise an error:

const firstnquery = uploadsRef.limitToFirst(limitToFirst);
const lastnquery = firstnquery.limitToLast(limitToLast);

Firebase queries are based on cursors. This means that to get the data for the next page, you must know the last item on the previous page. This is different from most databases, which work based on offsets. Firebase doesn't support offset based queries, so you'll need to know the value of createdBy and the key of the last item of the previous page.

With that, you can get the next page of items with:

admin.database().ref('uploads')
     .orderByChild('createdBy')
     .startAt(idOfLastItemOfPreviousPage, keyOfLastItemOfPreviousPage)
     .limitToFist(pageSize + 1)

I highly recommend checking out some other questions on implementing pagination on the realtime database, as there are some good examples and explanations in there too.

Failing to implement pagination in Firebase realtime database

1 Answers