How to remove duplicate objects in a List<MyObject> without equals/hashcode?

If for some reasons you don't want to override the equals method and you want to remove duplicates based on multiple properties, then we can create a generic method to do that.

We can write it in 2 versions:

1. Modify the original list:

@SafeVarargs
public static <T> void removeDuplicatesFromList(List<T> list, Function<T, ?>... keyFunctions) {

    Set<List<?>> set = new HashSet<>();

    ListIterator<T> iter = list.listIterator();
    while(iter.hasNext()) {
        T element = iter.next();

        List<?> functionResults = Arrays.stream(keyFunctions)
                .map(function -> function.apply(element))
                .collect(Collectors.toList());

        if(!set.add(functionResults)) {
            iter.remove();
        }
    }
}

2. Return a new list:

@SafeVarargs
public static <T> List<T> getListWithoutDuplicates(List<T> list, Function<T, ?>... keyFunctions) {

    List<T> result = new ArrayList<>();

    Set<List<?>> set = new HashSet<>();

    for(T element : list) {
        List<?> functionResults = Arrays.stream(keyFunctions)
                .map(function -> function.apply(element))
                .collect(Collectors.toList());

        if(set.add(functionResults)) {
            result.add(element);
        }
    }

    return result;
}

In both cases we can consider any number of properties.

For example, to remove duplicates based on 4 properties title, author, url and description:

removeDuplicatesFromList(blogs, Blog::getTitle, Blog::getAuthor, Blog::getUrl, Blog::getDescription);

The methods work by leveraging the equals method of List, which will check the equality of its elements. In our case the elements of functionResults are the values retrieved from the passed getters and we can use that list as an element of the Set to check for duplicates.

Complete example:

public class Duplicates {

    public static void main(String[] args) {

        List<Blog> blogs = new ArrayList<>();
        blogs.add(new Blog("a", "a", "a", "a"));
        blogs.add(new Blog("b", "b", "b", "b"));
        blogs.add(new Blog("a", "a", "a", "a"));    // duplicate
        blogs.add(new Blog("a", "a", "b", "b"));
        blogs.add(new Blog("a", "b", "b", "b"));
        blogs.add(new Blog("a", "a", "b", "b"));    // duplicate

        List<Blog> blogsWithoutDuplicates = getListWithoutDuplicates(blogs, 
                Blog::getTitle, Blog::getAuthor, Blog::getUrl, Blog::getDescription);
        System.out.println(blogsWithoutDuplicates); // [a a a a, b b b b, a a b b, a b b b]
        
        removeDuplicatesFromList(blogs, 
                Blog::getTitle, Blog::getAuthor, Blog::getUrl, Blog::getDescription);
        System.out.println(blogs);                  // [a a a a, b b b b, a a b b, a b b b]
    }

    private static class Blog {
        private String title;
        private String author;
        private String url;
        private String description;

        public Blog(String title, String author, String url, String description) {
            this.title = title;
            this.author = author;
            this.url = url;
            this.description = description;
        }

        public String getTitle() {
            return title;
        }

        public String getAuthor() {
            return author;
        }

        public String getUrl() {
            return url;
        }

        public String getDescription() {
            return description;
        }

        @Override
        public String toString() {
            return String.join(" ", title, author, url, description);
        }
    }
}

You can use distinct to remove duplicates

List<Blog> blogList = ....// add your list here

blogList.stream().distinct().collect(Collectors.toList());

This can be logically solved using a property. Here I have a property called a key.

1. Take out any String property in the object and put it in the list.
2. Check in the list weather that property contains if so then remove it.
3. Return the object list.

List<Object> objectList = new ArrayList<>();
 List<String> keyList = new ArrayList<>();
  objectList.forEach( obj -> {
   if(keyList.contains(unAvailabilityModel.getKey())) 
         objectList.remove(unAvailabilityModel); 
    else
        keyList.add(unAvailabilityModel.getKey();
});
return objectList;

We can also use Comparator to check duplicate elements. Sample code is given below,

private boolean checkDuplicate(List studentDTOs){

    Comparator<StudentDTO > studentCmp = ( obj1,  obj2)
            ->{
        if(obj1.getName().equalsIgnoreCase(obj2.getName())
                && obj1.getAddress().equalsIgnoreCase(obj2.getAddress())
                && obj1.getDateOfBrith().equals(obj2.getDateOfBrith())) {
            return 0;
        }
        return 1;
    };
    Set<StudentDTO> setObj = new TreeSet<>(studentCmp);
    setObj.addAll(studentDTOs);
    return setObj.size()==studentDTOs.size();
}

It is recommended to override equals() and hashCode() to work with hash-based collections, including HashMap, HashSet, and Hashtable, So doing this you can easily remove duplicates by initiating HashSet object with Blog list.

List<Blog> blogList = getBlogList();
Set<Blog> noDuplication = new HashSet<Blog>(blogList);

But Thanks to Java 8 which have very cleaner version to do this as you mentioned you can not change code to add equals() and hashCode()

Collection<Blog> uniqueBlogs = getUniqueBlogList(blogList);

private Collection<Blog> getUniqueBlogList(List<Blog> blogList) {
    return blogList.stream()
            .collect(Collectors.toMap(createUniqueKey(), Function.identity(), (blog1, blog2) -> blog1))
            .values();
}
List<Blog> updatedBlogList = new ArrayList<>(uniqueBlogs);

Third parameter of Collectors.toMap() is merge Function (functional interface) used to resolve collisions between values associated with the same key.

This question has already a great bunch of possible solutions. Not to mention it's age Normally I would recommend leaning on equals and hashCode, but today I came in the situation where that wasn't possible, and sice I came here via Google I'll share a solution with streams.

Since it's a list we may utilize .stream().filter() with with a custom predicate:

public static Predicate<Blog> distinctByKeys(Function<Blog, ?> ...keyExtractors) {
    final Map<Object, Boolean> seen = new ConcurrentHashMap<>();
    return t -> seen.putIfAbsent(Arrays.stream(keyExtractors)
            .map(keyExtractor -> keyExtractor.apply(t))
            .toList(), Boolean.TRUE) == null;
}

The predicate uses parts of Loris' answer.

To deduplicate a list of Blogs with java 17 syntax:

List<Blog> blogs = List.of(/*list of Blogs*/);

List<Blog> distinctList = blogs.stream()
    .filter(distinctByKeys(Blog::getTitle, Blog::getAuthor, Blog::getUrl, Blog::getDescription)
    .toList();

If you are not allow to modify class then better you use HashMap Structure should be like this Map<String, Object> = new HashMap<>();

1. create key by concatenating all required fields (your case key = title+author+url+description)
2. put into map
   • if key already exist ignore it
3. get values