How does groupingBy work when classifier returns a List?

As Hulk already said, there's absolutely no magic involved here. The List is compared with the equals method, just like any other object would.

To answer your second question, the type of items within a list can be controlled with generics. It's too bad they actually use a raw type in the answer you linked to. One should have used List<?> instead, but List<Object> would also have been fine.

Typically, the underlying array of a list implementation uses Object[] (at least for ArrayList is this the case).

A List<Object> can only contain Object instances, but guess what, all objects in the Java language are Objects, so that list can contain any object.

Immutability

It is important that the keys of a map (in this case List) are immutable. The way a typical Map works, is that it relies on the keys not being changed. Fortunately, the author of the linked post uses List::of, which returns an unmodifiable list.

Related: Are mutable hashmap keys a dangerous practice?

^{The author of the linked post used a List as key, in which he stored the fields to use as classifier. I would propose an alternative solution to using a list so select multiple fields to use as classifier. I would create a record instead, to make the intent clear of creating a classifier: record AgeAndNameClassifier(int age, String name) { } and then grouping by: groupingBy(item -> new AgeAndNameClassifier(item.getAge(), item.getName())...}

What is happening here is that the value returned by the classifier is compared using the equals()-method, which in case of a List is specified:

List.equals():

Returns true if and only if the specified object is also a list, both lists have the same size, and all corresponding pairs of elements in the two lists are equal.

Actually, the comparing and equals-invoking is done by the underlying Map implementation.

Your second question

How can List contain different variable types? Does the compiler treat it like a List<Object>?

Not quite, it is a raw type, see Java difference between List and List<Object> for some differences.

As for your third question,

Is it anti-pattern to use List?

This is answered here: ArrayList as key in HashMap

summary: it is risky, as is using mutable types as keys in general. hashCode must not change after insertion in a HashMap, and that means that neither the list itself nor any item in the list may be modified in a way that might change the results of hashCode() or equals().