I am trying to choose correct database and structure. Options of database: MongoDB or DynamoDB.
Options of document structure:
{
connected_from: string;
connected_to: Array<string>
}
Which if chosen will be used with MongoDB because I have to frequently search by connected_to items ($in). I need this for billions of documents and am opposed to the idea of sharding, because both fields will be indexed and might face memory issues. Example was Discord that made them switch to Cassanda which won't be an option in my case because it would require a secondary index which is not performant.
Second way is batchWrite with DynamoDB with this schema:
{
connected_from: string;
connected_to: string;
}
and write this document for every item in the connected_to array. Problems :
connected_toarray will possibly exceed the 16 item limit for batchwrite and I will have to loop over hundreds of elements which won't be performant.- Amount of write requests will make it expensive.
Would have been nice if DynamoDB had "$in" query without doing scan and implemented MongoDB structure. Will MongoDB be able to handle billions of docs and scale with this structure or is there a way to do this with DynamoDB or some other scalable database like Cassandra?