MongoDB how to achieve the same result as aggregate $lookup on a sharded collection

Viewed 29

Suppose you have the following setup in MongoDB:

db.createCollection("dataRows");

db.dataRows.insertMany([
    {
        "_id" : ObjectId("62d07ad59154a27f8b8dd601"),
        "name" : "Bob"
    }, {
        "_id" : ObjectId("62d07ad59154a27f8b8dd602"),
        "name" : "Roger"
    }, {
        "_id" : ObjectId("62d07ad59154a27f8b8dd603"),
        "name" : "Emanuel"
    }, {
        "_id" : ObjectId("62d07ad59154a27f8b8dd604"),
        "name" : "Smith"
    }
]);

db.createCollection("dataRowLists");

db.dataRowLists.insertMany([
    {
        "_id" : ObjectId("62d07ad29154a27f8b8dce01"),
        "dataRowIds" : [
            ObjectId("62d07ad59154a27f8b8dd601")
        ]
    }, {
        "_id" : ObjectId("62d07ad29154a27f8b8dce02"),
        "dataRowIds" : [
            ObjectId("62d07ad59154a27f8b8dd601"),
            ObjectId("62d07ad59154a27f8b8dd603"),
            ObjectId("62d07ad59154a27f8b8dd604")
        ]
    }
]);

Basically, there are 2 collections:

dataRows contains documents with some data, in this trivial example, just a name.

dataRowLists contains documents which, in turn, group the documents that are in the dataRows collection by using an array of IDs.

To fetch the data rows for a specific grouping you can run this query:

db.getCollection("dataRowLists").aggregate([
    { 
        $match: { 
            _id: ObjectId("62d07ad29154a27f8b8dce02") 
        } 
    },
    {
        $lookup: {
            from: "dataRows",
            localField: "dataRowIds",
            foreignField: "_id",
            as: "dataRows"
        }
    }
]);

The result is:

{ 
    "_id" : ObjectId("62d07ad29154a27f8b8dce02"), 
    "dataRowIds" : [
        ObjectId("62d07ad59154a27f8b8dd601"), 
        ObjectId("62d07ad59154a27f8b8dd603"), 
        ObjectId("62d07ad59154a27f8b8dd604")
    ], 
    "dataRows" : [
        {
            "_id" : ObjectId("62d07ad59154a27f8b8dd601"), 
            "name" : "Bob"
        }, 
        {
            "_id" : ObjectId("62d07ad59154a27f8b8dd603"), 
            "name" : "Emanuel"
        }, 
        {
            "_id" : ObjectId("62d07ad59154a27f8b8dd604"), 
            "name" : "Smith"
        }
    ]
}

As can be seen, this result contains an array with the data rows.

Unfortunately, if the data rows collection becomes sharded (which is the case for me), this query will not work.

Here is the command to make the data rows collection sharded in this trivial example:

sh.shardCollection("customer_a.dataRows", {_id: 1});

And, here is the error message:

"errmsg" : "customer_a.dataRows cannot be sharded"

Question: is there a workaround for this issue?

Please note: suppose there are a thousand data rows that are grouped, so fetching initially the array and then running a find $in should not be an option.

0 Answers
Related