I need to perform a $group operation over my entire collection. This group stage is reaching the limit of 100MB RAM usage.
The $group stage has a limit of 100 megabytes of RAM. By default, if the stage exceeds this limit, $group will produce an error. However, to allow for the handling of large datasets, set the allowDiskUse option to true to enable $group operations to write to temporary files.
I'm not limited by the RAM but I couldn't find how to increase this memory usage limit. Does anyone know how to config this restriction?
Setting up allowDiskUse to true will solve the solution but I assume the whole operation will be much slower and I'd like to find a better solution.
{
$group: {
_id: {
producer: "$producer",
dataset:"$dataset",
featureOfInterest:"$_id.featureOfInterest",
observedProperty:"$_id.observedProperty"
},
documentId: {$push:"$documentId"}
}
}
This $group operation is performed over entire complexe objects (producer and dataset). I understand that this operation is expensive since "It requires to scan the entire result set before yielding, and MongoDB will have to at least store a pointer or an index of each element in the groups." I'd rather $group on uniqueId fields for both of these object.
How could I $group object using unique ID and $project the whole object afterwards?
I'd like to obtain the same result as the group operation above using the group operation below at the beginning of my aggregation pipeline :
{
$group: {
_id: {
producer: "$producer.producerId",
dataset:"$dataset.datasetId",
featureOfInterest:"$_id.featureOfInterest",
observedProperty:"$_id.observedProperty"
},
documentId: {$push:"$documentId"}
}
}