Java 8:How to remove duplicates from the List based on multiple properties preserving the order

If you want to preserve order, obviously don't call .sorted, which messes with order.

More generally using streams here is complicated. Streams like it if the operation(s) you want to perform on each element within a stream are independent (do not need to look at anything except the one element being considered, i.e. no need to look at neighbours). That's not the case here.

If it is correct to remove any student with an attendence of 100 (by the way, that's a typo, the proper word is attendance), then all this stuff about 'duplicates' is a red herring, and all you need is:

list.removeIf(s -> s.getStudentAttendence() >= 100);

But if the idea is: Remove a record only if its attendence is 100+, and there is at least one other record in the list with the same name, it gets more complicated.

The primary issue is that your data storage mechanism is not in an appropriate form for this job. If you just stop with the lambdas, this is not hard. It helps to think about your list as consisting of 100 million entries. It is obviously not feasible for the entire stream op to keep the names of 100 million entries 'in memory'. You don't have that much memory. The datastructure (a List) also does not offer any fast lookups; there is no way to write code that answers the question 'how many records with studentName Sam is in this list?) without looping through 100 million entries which is a non-trivial job.

Thus, given the limits of:

• The input data is in List form.
• The input data is not already sorted.
• The output must keep the same order as the input.

Then the job is impossible on its face!

So, instead you need to accept that it isn't an easy one-liner, and that you need to first make alternative versions of the same data store that do store what you need.

Then there are additional concerns. In particular, what happens if you have 3 Sam students and each record has studentAttendence = 100? Should they ALL be deleted? Should none be deleted? Delete 2 arbitrary ones?

Often if you're having trouble writing an algorithm, the actual problem is that you haven't fully specified the behaviour you want, and thus your floundering is mostly down to you not fully understanding the problem, more than it is a coding issue.

Let's say the rule is simply: Delete all students with attendence = 100, but only if there is a record with the same name with an attendence below 100. If all records have an attendence of 100, keep them all, then:

List<Students> students = ...;
Set<String> dupeNames = students.stream()
  .filter(s -> s.getAttendence() < 100)
  .map(Student::getStudentName)
  .collect(Collectors.toSet());

students.removeIf(s -> s.getAttendence() < 100 && dupeNames.contains(s.getStudentName());

Will do the job, and will do it quickly. (O(n), to be algorithmically specific: Making a set-based duplicate requires constant-time steps per student record, so O(n), and the removeIf call similarly requires checking each student, but only having to do constant-time work per step, because .contains() on a set is constant time assuming good hash distribution, which Strings usually have), thus, a constant amount of O(n) operations means the whole operation is O(n): The time it takes grows linearly with how many students are in your input list (vs solutions that involve scanning the whole list every time you process a single entry in the list, which grows with the square of the input size).

Assuming studentId is uniqe as given in the example you coulduse the method List.removeIf with a BiPredicate accepting a student and the list of students.

BiPredicate<Student,List<Student>> pred = 
    (stud,list) -> list.stream()
                       .filter(s -> s.getStudentId() != stud.getStudentId())
                       .anyMatch(s -> s.getStudentName().equals(stud.getStudentName()) && stud.getStudentAttendence() == 100);

students.removeIf(stud -> pred.test(stud, students));

You should Filter record like this:

studentList.stream().filter(s -> s.getStudentAttendence() != 100)
                .filter(distinctByKeys(Student::getStudentName, Student::getStudentAttendence))
                .collect(Collectors.toList());

distinctByKeys method:

private static <T> Predicate<T> distinctByKeys(Function<? super T, ?>... keyExtractors) {
        final Map<List<?>, Boolean> seen = new ConcurrentHashMap<>();

        return t -> {
            final List<?> keys = Arrays.stream(keyExtractors).map(ke -> ke.apply(t)).collect(Collectors.toList());

            return seen.putIfAbsent(keys, Boolean.TRUE) == null;
        };
    }

After filtering

[Student [studentId=2, studentName=Sam, studentAttendence=50, studentAddress=New York], 
Student [studentId=3, studentName=Sam, studentAttendence=60, studentAddress=New York], 
Student [studentId=4, studentName=Nathan, studentAttendence=40, studentAddress=LA], 
Student [studentId=5, studentName=Ronan, studentAttendence=76, studentAddress=Atlanta]]

Try this:

Map<String, Integer> checkList = new HashMap<>();
Student [] buffer = studentsSupplier.get()
    .stream()
    .map( student -> 
    {
        checkList.compute( student.getStudentName(), (v,k) -> (v == null) ? Integer.valueOf( 1 ) : Integer.valueOf( v.intValue() + 1 ) );
        return student;
    }
    .toArray( Student []::new );
List<Student> result = buffer.stream()
    .filter( student -> (checkList.get( student.getStudentName() ).intValue() == 1) || (student.getAttendance() != 100) )
    .collect( Collectors.toList() );

This assumes that a record with an attendance of 100 remains in the list if it is the only one for that student.

The nasty version of the same code would look like this, but it is even less understandable:

Map<String, Integer> checkList = new HashMap<>();
List<Student> result = studentsSupplier.get()
    .stream()
    .peek( student -> checkList.compute( student.getStudentName(), (v,k) -> (v == null) ? Integer.valueOf( 1 ) : Integer.valueOf( v.intValue() + 1 ) )
    .collect( Collectors.toList() )
    .stream()
    .filter( student -> (checkList.get( student.getStudentName() ).intValue() == 1) || (student.getAttendance() != 100) )
    .collect( Collectors.toList() );