I am working on a project with collection that grow fast. the collection holds data on email tracking and email status(respond, bounced, spam,...) i merged them together to improve query performance. i have complex queries with $lookup and aggregate. my question is there any solution to manage the collection without archiving data. the collection can grow by million a day. in certain point even sharding wont help. how the big companies solve this problem. schema example
{
ownerId: mongoId,
threadId: 1234,
messageId: 23455,
subjectLine: hello,
emailState: 1,
created: timestamp,
update: timestamp,
count: 3,
opens: [
{
userAgent: firefox,
ip: 1.2.4,
created: timestamp
}
]
}