ASP.NET Core 5.0 and MongoDB - data model and key data type for large amount of data

Admin 1
Viewed 137

I don't actually try to do this, I'm just learning, but I would really like to know what would be the best solution, maybe there are some specialists here who are willing to help.

Let's take the most basic example: collection of posts, each post can contain collection of comments. And let's assume large amount of data.

public class Post
    {
        [BsonId]
        [BsonRepresentation(BsonType.ObjectId)]
        public Guid Id { get; set; }
        public string PostBody { get; set; }
    }

public class Comment
    {
        [BsonId]
        [BsonRepresentation(BsonType.ObjectId)]
        public Guid Id { get; set; }
        public Guid PostId { get; set; }
        public string CommentBody { get; set; }
    }

This model is not embedded, so it technically should be faster, because if you want to display posts only, there is no need to get comments. But if you want to get post and comments, you need to search whole comments collection.

And about key data type - int is too small, long int also will eventually run out. Guid is also finite, but should work for a little longer.

Is there optimal solution to this problem?

mongodb .net-core asp.net c#
1 Answers

This model is not embedded, so it technically should be faster

this is not correct. embedding is faster than doing joins/lookups. but there are downsides to embedding as well.

  • deeply nested/embedded items are harder to query and update.
  • need to use projections to exclude embedded items if they're not needed for a query.
  • the whole document cannot exceed 16mb. so not suitable for unbounded arrays.

on the other hand, if you go the referenced route and keep data in separate collections, you need to do lookups/joins which are slower than retrieving embedded documents.

But if you want to get post and comments, you need to search whole comments collection.

you don't have to search the whole collection because there's by default a unique index on the Id field of a collection. this index will be utilized when doing lookups/joins. and if lookups are initiated from the other side, you can easily create an index for the foreign field. example below.

int is too small, long int also will eventually run out. Guid is also finite

correct. that's why mongodb has ObjectId which works wonders with 99% of scenarios.

so... for your hypothetical scenario of posts & comments, you can actually embed the comments inside the post entity. because depending on the size of the comment body, you might be able to store a few hundred comments in a single post entity while keeping the total data size of the post under 16mb.

ideally, the embedding vs. referencing decision should be made according to the query patterns of your application while keeping the 16mb limit in mind.

here's an example console program using the MongoDB.Entities library (which i'm the author of) that simplifies much of the ceremony of the official driver. i've chosen the referenced route below.

using MongoDB.Driver;
using MongoDB.Entities;
using System.Threading.Tasks;

namespace TestApplication
{
    public class Post : Entity
    {
        public string Title { get; set; }
        public string Body { get; set; }
        public Many<Comment> Comments { get; set; }

        public Post()
            => this.InitOneToMany(() => Comments);
    }

    public class Comment : Entity
    {
        public One<Post> Post { get; set; }
        public string Body { get; set; }
    }

    public class PostWithComments : Post
    {
        public Comment[] CommentList { get; set; }
    }

    public static class Program
    {
        private static async Task Main()
        {
            await DB.InitAsync("test");

            //create some posts
            var post1 = new Post { Title = "post one", Body = "hellow world 1" };
            var post2 = new Post { Title = "post two", Body = "hellow world 2" };
            await new[] { post1, post2 }.SaveAsync();

            //create some comments
            var pst1comment1 = new Comment { Body = "post 1 comment 1", Post = post1 };
            var pst1comment2 = new Comment { Body = "post 1 comment 2", Post = post1 };
            var pst2comment1 = new Comment { Body = "post 2 comment 1", Post = post2 };
            var pst2comment2 = new Comment { Body = "post 2 comment 2", Post = post2 };
            await new[] { pst1comment1, pst1comment2, pst2comment1, pst2comment2 }.SaveAsync();

            //add the comments to the posts
            await post1.Comments.AddAsync(new[] { pst1comment1, pst1comment2 });
            await post2.Comments.AddAsync(new[] { pst2comment1, pst2comment2 });

            //retrieve 1st post only
            var p1 = await DB.Find<Post>()
                             .OneAsync(post1.ID);

            //retrieve comments of 1st post
            var p1comments = await DB.Find<Comment>()
                                     .Match(c => c.Post.ID == post1.ID)
                                     .ExecuteAsync();

            //retrieve 2nd post with comments
            var p2 = await DB.Fluent<Post>()
                             .Match(p => p.ID == post2.ID)
                             .Lookup<Post, Comment, PostWithComments>(
                                DB.Collection<Comment>(), //foreign collection to join
                                p => p.ID,                //local field
                                c => c.Post.ID,           //foreign field
                                pwc => pwc.CommentList)   //the field to store looked up comments
                             .ToListAsync();
        }
    }
}

create an index on the foreign field to make the lookups faster

await DB.Index<Comment>()
        .Key(c => c.Post.ID, KeyType.Ascending)
        .CreateAsync();
user_43684850
Related