How to map multiple properties under groupingBy property in java streams

You could rather prefer to choose the return type to be a Map<String, Map<String, Double>> or a custom class with appropriate equals and hashCode to ensure the uniqueness amongst the inner List<Custom>. I would frame the solution based on the former, and you can convert it to the one which is more readable to your actual code.

Once you have grouped each branch specific students, what you could do to ensure firstName is mapped to maximum average marks of that student is to perform a reduction using toMap with merge based on Double::max... and then collect these entries soted based on the marks (values).

Might look slightly complicated with the following code, but it could be broken into steps as well.

Map<String, LinkedHashMap<String, Double>> branchStudentsSortedOnMarks = stList.stream()
    .collect(Collectors.groupingBy(Student::getBranch, // you got it!
            Collectors.collectingAndThen(
                    Collectors.toMap(Student::getFirstName,
                            s -> s.getSubject().getAverageMarks(), Double::max), // max average marks per name
                    map -> map.entrySet().stream()
                            .sorted(Map.Entry.<String, Double>comparingByValue().reversed()) // sorted inn reverse based on value
                            .collect(Collectors.toMap(Map.Entry::getKey,
                                    Map.Entry::getValue, (a, b) -> b, LinkedHashMap::new)) 
            )));

Firstly, in your Map<String, List<Map<String,Double>>> the map inside the list would contain only one key-value pair. So I would suggest you to return Map<String, List<Entry<String, Double>>>. (Entry in java.util.Map)

Also, create a getAverageMarks in your student class which would return:

return subject.getAverageMarks();

// First define a function to sort based on average marks
UnaryOperator<List<Entry<String, Double>>> sort = 
    list -> {
        Collections.sort(list, Collections.reverseOrder(Entry.comparingByValue()));
        return list;
    };

// function to create entry
Function<Student, Entry<String, Double>> getEntry = 
    s -> Map.entry(s.getFirstName(), s.getAverageMarks());

// return this
list.stream()
    .collect(Collectors.groupingBy(
        Student::getBranch,        
        Collectors.mapping(getEntry, // map each student
                           // collect and apply sort as finisher
                           Collector.of(ArrayList::new,
                                        List::add,
                                        (x,y) -> {x.addAll(y); return x;},
                                        sort))));            

Input:

List<Student> stList = Arrays.asList(
        new Student("John", "Wall", "A", "a", "C", "sa", new SubjectMarks(65, 67, 100), LocalDate.now()),
        new Student("Arun", "Wall", "B", "a", "C", "sa", new SubjectMarks(45, 61, 95), LocalDate.now()),
        new Student("Marry", "Wall", "A", "a", "C", "sa", new SubjectMarks(90, 80, 92), LocalDate.now())
);

Idea:

1. group by "branch"
2. For each group (list) - sort by grade and map each student to a map of "name","grade".

Now it's easy to code:

Map<String, List<Map<String, Double>>> branchToSortedStudentsByGrade = stList.stream().collect(Collectors.groupingBy(
        Student::getBranch, Collectors.collectingAndThen(Collectors.toList(),
                l -> l.stream()
                        .sorted(Comparator.comparing(st -> st.getSubject().getAverageMarks(), Comparator.reverseOrder()))
                        .map(student -> Collections.singletonMap(student.getFirstName(), student.getSubject().getAverageMarks()))
                        .collect(Collectors.toList()))));

Output:

{
  A=[{Marry=87.0}, {John=77.0}],
  B=[{Arun=67.0}]
}

By the way:

Note that you divide by an integer in a floating-point context in getAverageMarks:

public double getAverageMarks() {
    double avg = (getBiology() + getMaths() + getComputers())/3;
    return avg;
}

This will cause all grades to be in this format- xx.0

If it's by mistake, I would recommend on this approach:

public double getAverageMarks() {
    return DoubleStream.of(maths, biology, computers)
            .average()
            .getAsDouble();
}