Is there a better way to work with Wildcards in Java?

The problem is the use of the constructor, which (naturally) has its very specific type, which then isn't compatible with the type of the variable where you're trying to store it. To make this explanation dig a little deeper into the fun and excitement of Java type wildcards, I did a type parameter rather than your Enum. (All of this is using Java's var, to keep the code slightly cleaner. If you're using Java11, this code will compile fine, except for the type errors and other warnings that I discuss below. Also, your original code has a problem with the Liskov Substitution Principle. I'll leave that alone for now and fix it below (4).

public interface Foo {
  class A { }
  class B extends A {}
  class C extends B {}

  class FooMap<T> {
    Map<? extends T, ? extends Set<? extends T>> f; // (1)
    List<? extends Set<? extends T>> fr;

    public FooMap(
        Map<? extends T, ? extends Set<? extends T>> f,
        List<? extends Set<? extends T>> fr) {
      this.f = f;
      this.fr = fr;
    }

    static void exercise1() {
      var hm = new HashMap<B, HashSet<B>>();
      var list = new ArrayList<HashSet<B>>(); // (2)
      var fm = new FooMap<A>(hm, list);
    }

    static void foo(
        List<? extends Set<? extends Integer>> listOfSetsOfInts) { }

    static void exercise2() {
      var diffSetType = new ArrayList<HashSet<Integer>>(); // (3)
      foo(diffSetType);
    }
  }
}

Comments:

(1) It's also possible to get rid of these wildcards. Here's some alternative code that will compile, but you'll get warnings about unchecked typecasts. Assuming you're treating the map and list as read-only, then you could substitute the code below. (Note: the Liskov Substitution Principle has things to say about this; see (4) below.) Once you allow for mutation, then things get more complicated.

    Map<T, Set<T>> f;
    List<Set<T>> fr;

    public FooMap(
        Map<? extends T, ? extends Set<? extends T>> f,
        List<? extends Set<? extends T>> fr) {
      this.f = (Map<T, Set<T>>) f;
      this.fr = (List<Set<T>>) fr;
    }

(2) The concrete type of list is ArrayList<HashSet<B>>, but we want to use it as if it's List<Set<A>>, which is allowed in the FooMap constructor's wildcards. Note that we're passing an explicit type parameter to the constructor to say that this is a FooMap<A>.

(3) The concrete type of diffSetType is ArrayList<HashSet<Integer>>, which is compatible with the argument type of foo().

(4) You need to think a little bit more carefully about the first type parameter of the HashMap, since we want to satisfy the Liskov Substitution Principle, which means satisfying the "producer extends, consumer super" (PECS) rule. The first parameter of the Map is a "consumer" type, so we really should write the code like this:

  class FooMap<T> {
    Map<T, Set<T>> f;
    List<Set<T>> fr;

    public FooMap(
        Map<? super T, ? extends Set<? extends T>> f,
        List<? extends Set<? extends T>> fr) {
      this.f = (Map<T, Set<T>>) f;
      this.fr = (List<Set<T>>) fr;
    }

    static void exercise1() {
      var hmA = new HashMap<A, HashSet<B>>();
      var hmB = new HashMap<B, HashSet<B>>();
      var hmC = new HashMap<C, HashSet<B>>();
      var list = new ArrayList<HashSet<B>>();
      var fm1 = new FooMap<A>(hmA, list); // okay
      var fm2 = new FooMap<B>(hmA, list); // okay
      var fm3 = new FooMap<C>(hmA, list); // type error (5)
      var fm4 = new FooMap<B>(hmB, list); // okay
      var fm5 = new FooMap<B>(hmC, list); // type error (6)
      var fm6 = new FooMap<C>(hmC, list); // type error (7)
    }

(5) HashSet<B> from list doesn't match ? extends Set<? extends C>

(6) HashMap<C, HashSet<B>> from hmC doesn't match Map<? super B, ? extends Set<? extends B>> because C doesn't match ? super B.

(7) hmC is HashMap<C, HashSet<B>>; HashSet<B> doesn't match ? extends Set<? extends C>

For your initial example, you don't need to go so far to support what you intend to support.

You have:

public FirstPair(Map<Enum, Set<Enum>> f, List<Set<Enum>> fr)

But Map and Set are interfaces, so if you intend to use this with a HashSet or a HashMap for the matter, you can, because those are classes that implement the Set and Map interface, respectively, and consequently adhere and respect the signature you defined for your constructor.

If you think in terms of interfaces common to the entities you want to support, it should simplify your life, and the signature of the constructor and/or methods/functions you are defining. If you come to think of it, typically if you want to allow other types to be used interchangeably, it is because you can treat them in a similar way, for the context of the functionality you are making available, so the interfaces tend to resolve that easily for you right away, and if there is no such interface out of the box for what you want to do, create one for the purpose. It will likely be easier to document, read and maintain in the long run. I hope that helps.