What are the production best practices to store a large number of document when using MongoDB?

user_2336115 0
Viewed 49

I am in need of storing applications transaction logs. Decided to use MongoDB. Every day there are almost 200000+- data is storing in single node MongoDB.

We have some reports and operation(if something happened then do something) depending on those logs. So, need to find documents matching different criteria. If going on that pace, is it vulnerable? Will it be slow to execute query?

Any suggestions to make it efficient to use MongoDB?

By the way, those data are in single collection. And MongoDB server version: 4.2.6

mongodb
1 Answers

mongo collections can grow to be many terabytes without much issue. to be able to query that data in a speedy manner, you will have to analyze your queries and create indexes for the fields that are used in those queries.

indexes are not free though. they will take both diskspace and use up RAM, because for indexes to be useful, they need to fit entirely in RAM.

in most cases, if indexes and collections grow beyond what your hardware can handle, you will have to archive/evict old data and trim down the collections.

if your queries need to include that evicted data in order to generate your reports, you will have to have another collection for summarized values/data of the evicted records which you will have to combine with present data when generating the reports.

alternatively sharding can help with big data but there are some limitations on queries you can do with sharded collections.

user_43684850
Related