Is it a good idea to store copies of documents from a mongodb collection in a dictionary list, and use this data instead of querying the database?

Viewed 96

I am currently developing a Python Discord bot that uses a Mongo database to store user data.

As this data is continually changed, the database would be subjected to a massive number of queries to both extract and update the data; so I'm trying to find ways to minimize client-server communication and reduce bot response times.

In this sense, is it a good idea to create a copy of a Mongo collection as a dictionary list as soon as the script is run, and manipulate the data offline instead of continually querying the database?

In particular, every time a data would be searched with the collection.find() method, it is instead extracted from the list. On the other hand, every time a data needs to be updated with collection.update(), both the list and the database are updated.

I'll give an example to better explain what I'm trying to do. Let's say that my collection contains documents with the following structure:

{"user_id": id_of_the_user, "experience": current_amount_of_experience}

and the experience value must be continually increased.

Here's how I'm implementing it at the moment:

online_collection = db["collection_name"] # mongodb cursor
offline_collection = list(online_collection.find()) # a copy of the collection

def updateExperience(user_id):

    online_collection.update_one({"user_id":user_id}, {"$inc":{"experience":1}})
    
    mydocument = next((document for document in offline_documents if document["user_id"] == user_id))
    mydocument["experience"] += 1

def findExperience(user_id):

    mydocument = next((document for document in offline_documents if document["user_id"] == user_id))
    return mydocument["experience"]

As you can see, the database is involved only for the update function.

Is this a valid approach? For very large collections (millions of documents) does the next () function have the same execution times or would there still be some slowdowns?

Also, while not explicitly asked in the question, I'd me more than happy to get any advice on how to improve the performance of a Discord bot, as long as it doesn't include using a VPS or sharding, since I'm already using these options.

1 Answers

I don't really see why not - as long as you're aware of the following :

  1. You will need the system resources to load an entire database into memory
  2. It is your responsibility to sync the actual db and your local store
  3. You do need to be the only person/system updating the database
  4. Eventually this pattern will fail i.e. db gets too large, or more than one process needs to update, so it isn't future-proof.

In essence you're talking about a caching solution - so no need to reinvent the wheel - many such products/solutions you could use.

It's probably not the traditional way of doing things, but if it works then why not

Related