implementing a pagging for huge dataset avoiding skip on mongo

user_13702631 0
Viewed 97

I am new in mongo world, and I would like to optimize my query cves = colCVE.find({cpe_searchField: {"$regex": cpe_regex}}).sort("Modified", -1)limit(limit).skip(skip).allow_disk_use(True) in order to avoid .skip since some blogs says when the data grows up then consume a lot of memory, and I would like to understand how to compute over _id and make quicker pagination over more than 300k objects. the below example is the search function that searches over a regex to match only the specific things instead of all rows.

def cvesForCPE(cpe, vulnProdSearch=False):
    if not cpe:
        return []

    cpe_regex = cpe
    final_cves = []
    cpe_searchField = (
        "vulnerable_product" if vulnProdSearch else "vulnerable_configuration"
    )


    # default strict search
    cves = colCVE.find({cpe_searchField: {"$regex": cpe_regex}}).sort("Modified", -1).allow_disk_use(True)
    final_cves = cves

    final_cves = sanitize(final_cves)
    return final_cves
pymongo mongodb pagination
1 Answers

If you sort on a field, and the field has distinct values, you can specify conditions based on that field to get the next page of results.

For example, supposing you have a field M:

  • Find all documents, sort by M, limit to one page.
  • Get the results (this is first page).
  • Suppose m1 is the value of M in the last document.
  • Find all documents where M > m1, sort by M, limit to one page.
  • Get the results (this is the second page).
  • etc.

This doesn't work when the values in M are not distinct. If you expect the number of adjacent duplicate values to be less than page size you can find where M >= m1 and remove documents from the beginning that you've already shown on the previous page.

user_95409250
Related