How to fetch unlimited documents using Mango query couchDB without providing "limit" field?

user_8159832 0
Viewed 2204

I am using Mango query in couch db for almost every query to find the documents. Here I am facing a problem in fetching all the documents matching the given conditions. The problem is that the default limit of mango query is 25 (means fetch 25 documents per query) and there are lot many documents in my database and I don’t have the exact count of documents. I cannot hard code the limit in mango query as I don’t know the upper limit of documents and I don’t think hard coding the limit is a good idea. Can anyone please help me with this issue? How can I make the limit as unlimited or is there any other way to handle this situation?

couchdb-mango couchdb
2 Answers

I countered the same issue and resolved it with a recursive function

const batchSize = 25;
let batchCount = 0;
let searchResults = [];

//  in my case I want to find docs with a certain date, hardcoded here as example
let selectedDate = '2017/10/30'; 

// the recursive function
let search = function (count, limit){
    return db.find({
        selector: {
            date: selectedDate  
        },
        limit: batchSize,
        skip: batchCount*batchSize 
    }).then(function(batch){
        if (batch.docs.length === 0){
            // there are no (more) search results, so we can return what we have

            return searchResults

        } else {
            // there may be more search results, so we add this batch to the result and call the function again to ask more. We increase the counter so that the first batch(es) are skipped

            for (var i = 0; i < batch.docs.length; i++) {
                searchResults.push(batch.docs[i])
            }
            batchCount ++
            return search(batchCount, batchSize) 
        }
    })
}

search(batchCount, batchSize).then(function(result){
    // you can handle the searchresults

})
user_76007660

Apparently there is a bookmark key that is returned on each _find call that you can then pass back to another _find call to get the next 'page' of results (assuming nothing else about the second query was altered from the first).

Not sure how practical this approach is if the previous query isn't guaranteed to be the same though.

I actually ended up using an iterative version of the @Erik's answer.

See the JSON request options in the docs here: https://docs.couchdb.org/en/latest/api/database/find.html

user_60325830
Related