How can I delete duplicates in a Dart List? list.distinct()?

Question

How do I delete duplicates from a list without fooling around with a set? Is there something like list.distinct()? or list.unique()?

void main() {
  print("Hello, World!");

  List list = ['abc',"abc",'def'];
  list.forEach((f) => print("this is list $f"));

  Set set = new Set.from(list);
  print("this is #0 ${list[0]}");
  set.forEach((f) => print("set: $f"));

  List l2= new List.from(set);
  l2.forEach((f) => print("This is new $f"));
}

Hello, World!
this is list abc
this is list abc
this is list def
this is #0 abc
set: abc
set: def
This is new abc
This is new def

Set seems to be way faster!! But it loses the order of the items :/

user_1058292 · Answer

Use toSet and then toList

  var ids = [1, 4, 4, 4, 5, 6, 6];
  var distinctIds = ids.toSet().toList();

Result: [1, 4, 5, 6]

Or with spread operators:

var distinctIds = [...{...ids}];

user_940300 · Answer

I didn't find any of the provided answers very helpful. Here is what I generally do:

final ids = Set();
myList.retainWhere((x) => ids.add(x.id));

Of course you can use any attribute which uniquely identifies your objects. It doesn't have to be an id field.

Benefits over other approaches:

Preserves the original order of the list
Works for rich objects not just primitives/hashable types
Doesn't have to copy the entire list to a set and back to a list

Update 09/12/21
You can also declare an extension method once for lists:

extension Unique on List {
  List unique([Id Function(E element)? id, bool inplace = true]) {
    final ids = Set();
    var list = inplace ? this : List.from(this);
    list.retainWhere((x) => ids.add(id != null ? id(x) : x as Id));
    return list;
  }
}

This extension method does the same as my original answer. Usage:

// Use a lambda to map an object to its unique identifier.
myRichObjectList.unique((x) => x.id);
// Don't use a lambda for primitive/hashable types.
hashableValueList.unique();

user_10914124 · Answer

Set works okay, but it doesn't preserve the order. Here's another way using LinkedHashSet:

import "dart:collection";

void main() {
  List arr = ["a", "a", "b", "c", "b", "d"];
  List result = LinkedHashSet.from(arr).toList();
  print(result); // => ["a", "b", "c", "d"]
}

https://api.dart.dev/stable/2.4.0/dart-collection/LinkedHashSet/LinkedHashSet.from.html

user_10586776 · Answer

Try the following:

List duplicates = ["a", "c", "a"];

duplicates = duplicates.toSet().toList();

Check this code on Dartpad.

user_4079317 · Answer

If you want to keep ordering or are dealing with more complex objects than primitive types, store seen ids to the Set and filter away those ones that are already in the set.

final list = ['a', 'a', 'b'];
final seen = Set();
final unique = list.where((str) => seen.add(str)).toList();

print(unique); // => ['a', 'b']

user_8893132 · Answer

I am adding this to atreeon's answer. For anyone that want use this with Object:

class MyObject{
  int id;

  MyObject(this.id);


  @override
  bool operator ==(Object other) {
    return other != null && other is MyObject && hashCode == other.hashCode;
  }


  @override
  int get hashCode => id;
}

main(){
   List list = [MyObject(1),MyObject(2),MyObject(1)];

   // The new list will be [MyObject(1),MyObject(2)]
   List newList = list.toSet().toList();
}

user_15269544 · Answer

//This easy way works fine

List myArray = [];
myArray = ['x', 'w', 'x', 'y', 'o', 'x', 'y', 'y', 'r', 'a'];

myArray = myArray.toSet().toList();

print(myArray);

// result => myArray =['x','w','y','o','r', 'a']

user_3146563 · Answer

Using Dart 2.3+, you can use the spread operators to do this:

final ids = [1, 4, 4, 4, 5, 6, 6];
final distinctIds = [...{...ids}];

Whether this is more or less readable than ids.toSet().toList() I'll let the reader decide :)

user_538284 · Answer

For distinct list of objects you can use Equatable package.

Example:

// ignore: must_be_immutable
class User extends Equatable {
  int id;
  String name;

  User({this.id, this.name});

  @override
  List

user_1798328 · Answer

Here it is, a working solution:

var sampleList = ['1', '2', '3', '3', '4', '4'];
//print('original: $sampleList');
sampleList = Set.of(sampleList).toList();
//print('processed: $sampleList');

Output:

original: [1, 2, 3, 3, 4, 4]
processed: [1, 2, 3, 4]

user_3411681 · Answer

Using the fast_immutable_collections package:

[1, 2, 3, 2].distinct();

Or

[1, 2, 3, 2].removeDuplicates().toList();

Note: While distinct() returns a new list, removeDuplicates() does it lazily by returning an Iterable. This means it is much more efficient when you are doing some extra processing. For example, suppose you have a list with a million items, and you want to remove duplicates and get the first five:

// This will process five items:
List newList = list.removeDuplicates().take(5).toList();

// This will process a million items:
List newList = list.distinct().sublist(0, 5);

// This will also process a million items:
List newList = [...{...list}].sublist(0, 5);

Both methods also accept a by parameter. For example:

// Returns ["a", "yk", "xyz"]
["a", "yk", "xyz", "b", "xm"].removeDuplicates(by: (item) => item.length);

If you don't want to include a package into your project but needs the lazy code, here it is a simplified removeDuplicates():

Iterable removeDuplicates(Iterable iterable) sync* {
  Set items = {};
  for (T item in iterable) {
    if (!items.contains(item)) yield item;
    items.add(item);
  }
}

Note: I am one of the authors of the fast_immutable_collections package.

user_1556386 · Answer

Remove duplicates from a list of objects:

class Stock {
  String? documentID; //key
  Make? make;
  Model? model;
  String? year;

  Stock({
    this.documentID,
    this.make,
    this.model,
    this.year,
  });
}

List of stock, from where we want to remove duplicate stocks

List stockList = [stock1, stock2, stock3];

Remove duplicates

final ids = stockList.map((e) => e.documentID).toSet();
stockList.retainWhere((x) => ids.remove(x.documentID));

user_7733170 · Answer

As for me, one of the best practices is sort the array, and then deduplicate it. The idea is stolen from low-level languages. So, first make the sort by your own, and then deduplicate equal values that are going after each other.

// Easy example
void dedup(List list, {removeLast: true}) {
  int shift = removeLast ? 1 : 0;
  T compareItem;
  for (int i = list.length - 1; i >= 0; i--) {
    if (compareItem == (compareItem = list[i])) {
      list.removeAt(i + shift);
    }
  }
}

// Harder example
void dedupBy(List list, I Function(T) compare, {removeLast: true}) {
  int shift = removeLast ? 1 : 0;
  I compareItem;
  for (int i = list.length - 1; i >= 0; i--) {
    if (compareItem == (compareItem = compare(list[i]))) {
      list.removeAt(i + shift);
    }
  }
}


void main() {
  List> list = [[1], [1], [2, 1], [2, 2]];
  print('$list');
  dedupBy(list, (innerList) => innerList[0]);
  print('$list');

  print('
 removeLast: false');

  List> list2 = [[1], [1], [2, 1], [2, 2]];
  print('$list2');
  dedupBy(list2, (innerList) => innerList[0], removeLast: false);
  print('$list2');
}

Output:

[[1], [1], [2, 1], [2, 2]]
[[1], [2, 1]]

removeLast: false
[[1], [1], [2, 1], [2, 2]]
[[1], [2, 2]]

user_14000604 · Answer

This is another way...

final reducedList = [];

list.reduce((value, element) {
    if (value != element) 
        reducedList.add(value);
    return element;
});

reducedList.add(list.last);

print(reducedList);

user_9221756 · Answer

List bigList = [];
List newList = [];  

for (var element in bigList) {
      var list = newList.where((i) => i.type == element.type).toList();
      if(list.isEmpty){
       newList.add(element);
      }
    }

user_13527782 · Answer

This is my solution

    List removeDuplicates(List list, IsEqual isEqual) {
      List output = [];
      for(var i = 0; i < list.length; i++) {
        bool found = false;
        for(var j = 0; j < output.length; j++) {
          if (isEqual(list[i], output[j])) {
            found = true;
          }
        }
        if (found) {
          output.add(list[i]);
        }
      }

      return output;
    }

Use it like this:

  var theList = removeDuplicates(myOriginalList, (item1, item2) => item1.documentID == item2.documentID);

or...

  var theList = removeDuplicates(myOriginalList, (item1, item2) => item1.equals(item2));

or...

user_17631015 · Answer

It works for me.

var list = [
 {"id": 1, "name": "Joshua"},
 {"id": 2, "name": "Joshua"},
 {"id": 3, "name": "Shinta"},
 {"id": 4, "name": "Shinta"},
 {"id": 5, "name": "Zaidan"}
];
list.removeWhere((element) => element.name == element.name.codeUnitAt(1));
list.sort((a, b) => a.name.compareTo(b.name));

Output:

[{"id": 1, "name": "Joshua"}, 
{"id": 3, "name": "Shinta"}, 
{"id": 5, "name": "Zaidan"}]

How can I delete duplicates in a Dart List? list.distinct()?

19 Answers