Remove duplicates from an array of objects based on the first 3 words of object property

user_8870206 0
Viewed 216

I have this array of objects which have a lot of duplicate entries. I can clean the array and get rid of the duplicate ones but the catch is I need to remove those which matches based on the property's first 3 words.

Suppose this is the array:

let arr = [
     {
         text: "Be good and you will be lonely. But there’s nothing wrong with being lonely.",
         id: 1
     },
     {
         text: "Coffee is a way of stealing time.",
         id: 2
     },
     {
         text: "Be good and you will be lonely. But there’s nothing wrong with being lonely.",
         id: 3
     }
];

I want to match the first 3 words of each of the texts and if it is a match then remove one of the matched objects from the old array and push the removed one to a new array.

So far I could remove the duplicate ones with this piece of code and I don't know what I should do next.

let texts     = {};

arr = arr.filter(function(currentObject) {
      if (currentObject.text in seenNames) {
           return false;
      } else {
           seenNames[currentObject.text] = true;
           return true;
      }
});

It would be a big help if someone point me to the right direction.

UPDATE:

I started the whole thing again with a different approach than before. As @Andreas and @freedomn-m said, I split the items based on the first 3 words and then tried to filter the original array by matching the split items. But right now I'm getting all the values back without any filtration.

let arr = [{
    "text": "Be good and you will be lonely. But there’s nothing wrong with being lonely.",
    "id": 1
  },
  {
    "text": "Coffee is a way of stealing time.",
    "id": 2
  },
  {
    "text": "Be good and you will be lonely. But there’s nothing wrong with being lonely.",
    "id": 3
  }
];

let removedItems = [];


let filtered = arr.filter((item, index) => {
  let splitItem = item["text"].split(" ").slice(0, 3).join(" ").toLowerCase();

  if (item["text"].toLowerCase().startsWith(splitItem, index + 1)) {
    return item;
  } else {
    removedItems.push(item);
  }

});



console.log(filtered);
console.log(removedItems);

arrays jquery javascript
1 Answers

For the original question of how to get 3 words, one option is to use .split() .slice() and .join():

var firstWords = item["text"].split(" ").slice(0, 3).join(" ");

you can then do a straight replacement of currentObject.text with firstWords, from the original question:

let texts = {};
arr = arr.filter(function(currentObject) {
    if (firstWords in seenNames) {
         return false;
    } else {
         seenNames[firstWords] = true;
         return true;
    }
});

The update attempts this, but has 2 issues:

  • .filter(function(item)) must return true/false (as it did originally) not the item/nothing.

  • item["text"].toLowerCase().startsWith(splitItem) will always be true as splitItem is built from item["text"]

Adding the removedItems additional list to the original gives:

let arr = [{
    "text": "Be good and you will be lonely. But there’s nothing wrong with being lonely.",
    "id": 1
  },
  {
    "text": "Coffee is a way of stealing time.",
    "id": 2
  },
  {
    "text": "Be good and you will be lonely. But there’s nothing wrong with being lonely.",
    "id": 3
  }
];

let removedItems = [];
let seenNames = {};

let filtered = arr.filter((item, index) => {
  let splitItem = item["text"].split(" ").slice(0, 3).join(" ").toLowerCase();
  
  if (splitItem in seenNames) {
    // already exists, so don't include in filtered, but do add to removed
    removedItems.push(item);
    return false;
  }
   
  // doesn't exist, so add to seen list and include in filtered
  seenNames[splitItem] = true;
  return true;
});

console.log(filtered);
console.log(removedItems);

user_21815140
Related