I need to extract data in the form of key-value pairs from a collection of records and merge them into their parent record in mongoDB

Viewed 165

Below I have a structure for supporting custom picklist fields (in this example) within my sails.js application. The general idea is we support a collection of custom picklist values on any model within the app and the customer can have total control of the configuration of the custom field.

I went with this relationship model as using a simple json field lacks robustness when it comes to updating each individual custom picklist value. If I allow a customer to change "Internal" to "External" I need to update all records that have the value "Internal" recorded against that custom picklist with the new value.

This way - when I update the "value" field of CustomPicklistValue wherever that record is referenced via ID it will use the new value.

enter image description here

Now the problem comes when I need to integrate this model into my existing report engine...

rawCollection
        .aggregate(
            [
                {
                    $match: {
                        createdAt: {
                            $gte: rangeEndDate,
                            $lte: rangeStartDate
                        },
                        ...$match
                    }
                },
                {
                    $project: {
                        ...$project,
                        total: $projectAggregation
                    }
                },
                {
                    $group: {
                        _id: {
                            ...$groupKey
                        },
                        total: {
                            [`$${aggrAttrFunc}`]: "$total"
                        }
                    }
                }
            ],
            {
                cursor: {
                    batchSize: 100
                }
            }
        )

Here is the main part of a method for retrieving and aggregating any models stored in my mongodb instance. A user can specify a whole range of things including but not limited to the model, field specific date ranges and filters such as "where certificate status equals expired" etc.

So I'm now presented with this data structure:

{
    id: '5e5fb732a9422a001146509f',
    customPicklistValues: [
        {
            id: '5e4e904f16ab94bff1a324a0',
            value: 'Internal',
            fieldName: 'Business Group',
            customPicklist: '109c7a1a9d00b664f2ee7827'
        },
        {
            id: '5e4e904f16ab94bff1a324a4',
            value: 'Slack',
            fieldName: 'Application',
            customPicklist: '109c5a1a9d00b664f2ee7827'
        }
    ],
}

And for the life of me can't work out if there's any way I can essentially pull out fieldName and value for each of the populated records as key-value pairs and add each to the parent record before running my match clause...

I think I need to use lookup to initially populate the customPicklistValues and then merge them somehow?

Any help appreciated.

EDIT:

@whoami has suggested I use $addFields. There was a fair amount I needed to do before $addFields to populate the linked records (due to how Waterline via sails.js handles saving Mongo ObjectIDs in related collections as strings), you can see my steps in compass here:

enter image description here

Final step would be to edit this or add a stage to it to actually be able to support a key:value pair like Business Group: "Finance" in this example.

1 Answers

You can try these stages after your $lookup stage :

db.collection.aggregate([
    {
        $addFields: {
            customPicklistValues:
            {
                $arrayToObject: {
                    $map: {
                        input: '$customPicklistValues',
                        in: { k: '$$this.fieldName', v: '$$this.value' }
                    }
                }
            }
        }
    },
    { $replaceRoot: { newRoot: { $mergeObjects: ['$customPicklistValues', '$$ROOT'] } } },
    { $project: { customPicklistValues: 0 } }
])

Test : MongoDB-Playground

Related