Use Comparator with TreeSet

As commented by Johannes Kuhn, you can get your desired behavior by using a NavigableSet (or SortedSet). No need to invent your own class.

Implementations of NavigableSet such as TreeSet may offer a constructor taking a Comparator object. That Comparator is used for sorting the elements of the set.

To our point here in this Question, that Comparator is also used in deciding to admitting new distinct elements rather than using the elements’ own Object#equals method.

And since there is no hashing involved in a TreeSet, there is no concern about overriding hashCode.

We can easily define our Comparator implementation. For convenience, we can call Comparator.comparing to make a comparator implementation. We define the comparator by passing a method reference for the getter method of your desired name field: User :: name.

You can add more criteria to your comparator by calling thenComparing. I leave that as an exercise for the reader.

For brevity, let's define your User class as a record. We simply declare the type and name of member fields. The compiler implicitly creates the constructor, getters, equals & hashCode, and toString.

record User( String name , String email , int age ) { }

Make some sample data.

List < User > listOfUsers =
        List.of(
                new User( "Bob" , "bob@x.com" , 7 ) ,
                new User( "Alice" , "alice@x.com" , 42 ) ,
                new User( "Carol" , "carol@x.com" , 77 )
        );

Define our set, a TreeSet.

NavigableSet < User > setOfUsers = new TreeSet <>( Comparator.comparing( User :: name )  );

Populate our set with 3 elements. Verify 3 elements by dumping to console.

setOfUsers.addAll( listOfUsers );
System.out.println( setOfUsers.size() + " elements in setOfUsers = " + setOfUsers );

Now we try to add another user with the same name but different values in the other fields.

setOfUsers.add( new User( "Alice" , "a@aol.com" , -666  ) );

By default, a record decides on equality by comparing each and every member field. So:

• If we have failed in our goal of using only name for comparison, we would get 4 elements in this set.
• If we have succeeded in using only name, then we should get 3 elements after having blocked admission of this interloper.

Dump to console.

System.out.println( setOfUsers.size() + " elements in setOfUsers = " + setOfUsers );

3 elements in setOfUsers = [User[name=Alice, email=alice@x.com, age=42], User[name=Bob, email=bob@x.com, age=7], User[name=Carol, email=carol@x.com, age=77]]

3 elements in setOfUsers = [User[name=Alice, email=alice@x.com, age=42], User[name=Bob, email=bob@x.com, age=7], User[name=Carol, email=carol@x.com, age=77]]

We see in those results (a) sorting of the elements by name, and (b) Blocking of the second Alice, with the original Alice remaining.

To see the alternate behavior, replace the setOfUsers definition with this:

Set < User > setOfUsers = new HashSet <>();

Running that edition of the code results in setOfUsers.size() being:

3 elements in setOfUsers = [User[name=Bob, email=bob@x.com, age=7], User[name=Carol, email=carol@x.com, age=77], User[name=Alice, email=alice@x.com, age=42]]

4 elements in setOfUsers = [User[name=Bob, email=bob@x.com, age=7], User[name=Carol, email=carol@x.com, age=77], User[name=Alice, email=alice@x.com, age=42], User[name=Alice, email=a@aol.com, age=-666]]

We see in those results (a) no particular sorting, and (b) the addition of a second "Alice", having increased the set from 3 elements to 4.

Caveat

One possible downside to my solution here is that we are violating the recommendation of the Javadoc of TreeSet to be “consistent with equals”, thereby violating the general contract of Set.

I am not sure if that issue is problematic or not — I do not have enough perspective to form a judgement.

Clean and Efficient

I would like to have UserNameSet which would allow to contain only users which have different name

You can apply composition and create a class that would maintain a Map and delegate all calls to it.

The approach of encapsulating a collection and providing limited access to it is more flexible and less tedious than extending a collection, and also doesn't create a tight coupling.

Similar advice, and also examples of disadvantages that come with making your code dependent on the existing collection by extending it, you can find in the book "Effective Java" by Joshua Bloch, item Favor composition over inheritance.

That's how such a class might look like:

class UserNameSet {
    private Map<String, User> userByName = new HashMap<>();
    
    public User add(User user) {
        return userByName.put(user.getName(), user); // or `putIfAbsent()` if you want to retain the previously added user
    }
    
    public User remove(User user) {
        return userByName.remove(user.getName());
    }
    
    public boolean contains(User user) {
        return userByName.containsValue(user.getName());
    }
    
    public User remove(String name) {
        return userByName.remove(name);
    }
    
    public boolean contains(String name) {
        return userByName.containsKey(name);
    }
    
    // all other methods that are required
}

We can also make this class to be generic and capable of wrapping any object.

For that, we need to introduce an additional argument - a function that will be responsible for extracting the target property from the object.

class MyCustomSet<K, V> {
    private Map<K, V> userByName = new HashMap<>();
    private Function<V, K> keyExtractor;
    
    public MyCustomSet(Function<V, K> keyExtractor) {
        this.keyExtractor = keyExtractor;
    }
    
    public V add(V user) {
        return userByName.put(keyExtractor.apply(user), user); // or `putIfAbsent()` if you want to retain the previously added user
    }
    
    public V remove(V user) {
        return userByName.remove(keyExtractor.apply(user));
    }
    
    public boolean contains(V user) {
        return userByName.containsValue(user);
    }
    
    public V removeByKey(K name) {
        return userByName.remove(name);
    }
    
    public boolean containsKey(K name) {
        return userByName.containsKey(name);
    }
    
    // all other methods that are required
}

And that's how it would be instantiated in the client code:

MyCustomSet<String, User> uniqueNameUsers = new MyCustomSet<>(User::getName);

Concise and Simple

The approach of utilizing sorted collections like TreeSet described in the Basil's Bourque answer can be used for a relatively small number of objects.

It's important to emphasize that storing the data in sorted collections has a cost. The larger they become, the slower they operate. TreeSet is backed by the Red-black tree and most of the actions like basic add(), remove() and contains() operations require O(n) time, apart from the edge-cases when we deal with the lowest/highest key.

commons-collections already provides an Equator interface to do what you suggest:

public interface Equator<T> {
    boolean equate(T o1, T o2);
    int hash(T o);
}

However direct support for creating collections based on an equator is limited. There are a few operations involving an equator available in CollectionUtils.

However you could leverage Transformer to wrap your desired objects into ones that use an equator and then use all the support commons-collections provides for transformers. For example using SetUtils.transformedSet:

class EquatorWrapper<T> {
    private final Class<T> clazz;
    private final T wrapped;
    private final Equator<T> equator;

    public EquatorWrapper(Class<T> clazz, T wrapped, Equator<T> equator) {
        this.clazz = clazz;
        this.wrapped = wrapped;
        this.equator = equator;
    }

    @Override
    public boolean equals(Object obj) {
        if (clazz.isInstance(obj)) {
            return equator.equate(wrapped, clazz.cast(obj));
        }
        return false;
    }

    @Override
    public int hashCode() {
        return equator.hash(wrapped);
    }
}

class EquatorTransformer<T> implements Transformer<T, Object> {
    private final Class<T> clazz;
    private final Equator<T> equator;

    public EquatorTransformer(Class<T> clazz, Equator<T> equator) {
        this.clazz = clazz;
        this.equator = equator;
    }
        
    @Override
    public Object transform(T input) {
        return new EquatorWrapper<>(clazz, input, equator);
    }
}

SetUtils.transformedSet(someSet, EquatorTransformer.of(someEquator, SomeClazz.clazz));