How do I remove duplicates in mongodb based on multiple fields?

You can do it in two ways

1. In one DB call : Using aggregation operator $out, Maybe you can also use $merge but which is not much useful for your case.
2. In two DB calls : As if you think $out is destructive and with millions of documents in collection can be an issue in production environment, then you can first read all the _id's of documents to be deleted & use .deleteMany() to delete all docs at once. (You can use any unique identifier on a doc instead of _id but I've used _id as it's indexed by default - which can help to run deleteMany() quicker).

Step 1:

Using $out - So as I've said it is destructive cause it will override the entire collection if input name matches or will create a new collection by the result of your aggregation query. So test your aggregation query very well prior to using $out as last stage. Also write data to temporary collection & rename the collections after everything is good enough. Consider a down-time while renaming collections

Query :

db.collection.aggregate([
  {
    $group: { _id: { name: "$name", value: "$value" },
      doc: { $last: "$$ROOT" } // Retrieve only last doc in a group
    }
  },
  {
    $replaceRoot: { newRoot: "$doc" } // replace doc as object as new root of document
  },
  { $out : 'collection_new' } // Test above aggregation & then use this 
])

Test : mongoplayground

Step 2:

1. Using an aggregation query you'll get list of _ids to be deleted from collection.

Query :

db.collection.aggregate([
    /**
     * Group on matching docs :
     * { name: "duplicate", value: false}, 
     * { name: "duplicate", value: true}, 
     * { name: "duplicate-yes", value: true},
     * { name: "notDuplicate", value: true} 
     * */
    {
      $group: {
        _id: { name: "$name", value: "$value" },
        _idsNeedsToBeDeleted: { $push: "$$ROOT._id" } // push all `_id`'s to an array
      }
    },
    /** Remove first element - which is removing a doc */
    {
      $project: {
        _id: 0,
        _idsNeedsToBeDeleted: { $slice: [ "$_idsNeedsToBeDeleted", 1, { $size: "$_idsNeedsToBeDeleted" } ] }
      }
    },
    {
      $unwind: "$_idsNeedsToBeDeleted" // Unwind `_idsNeedsToBeDeleted`
    },
    /** Group without a condition & push all `_idsNeedsToBeDeleted` fields to an array */
    {
      $group: { _id: "", _idsNeedsToBeDeleted: { $push: "$_idsNeedsToBeDeleted" } }
    },
    {$project : { _id : 0 }} // Optional stage
     /** At the end you'll have an [{ _idsNeedsToBeDeleted: [_ids] }] or [] */
  ])

Test : mongoplayground

2. Now using .deleteMany() - delete all docs :

Query :

db.collection.deleteMany( { "_id" : {$in : [_ids]} } );

Consideration prior to .deleteMany() you need to check aggregation result is not an empty array [] & has a doc with _idsNeedsToBeDeleted field which is an array. Also since we're matching against _id in DB - aggregations _idsNeedsToBeDeleted array will be an array of strings - So iterate over array, convert string to ObjectId() & use that array of ObjectId()'s in delete query.

Note :

Irrespective of what step you choose - Since we're grouping on name + value you need to make sure all of your docs has those fields.

i'm not sure about mongo but using the node you can remove duplicates. i tried this method for one of requirement that worked fine. please try this by chagning as you need var names.

function arrUnique(arr) {
    var cleaned = [];
    arr.forEach(function(itm) {
        var unique = true;
        cleaned.forEach(function(itm2) {
            if (_.isEqual(itm, itm2)) unique = false;
        });
        if (unique)  cleaned.push(itm);
    });
    return cleaned;
}

var newField = arrUnique(newField);

Doing it natively in node seemed to do the trick (probably not the fastest or most efficient way but the following worked):

const array = await db.collection(collectionName).find({}).toArray();
const newArr = array.map((item) => {
  const newObj = Object.assign({}, item, {
    hiWorld: `${item.name}${item.amount}`,
  });
  return newObj;
});
var uniqueItems = [];
var duplicateIds = [];
newArr.forEach((item) => {
  if (uniqueItems.includes(item.hiWorld)) {
    duplicateIds.push(item._id);
  } else {
    uniqueItems.push(item.hiWorld);
  }
});

await db.collection(collectionName).deleteMany({ _id: { $in: duplicateIds } });