Extend an existing stream collector instance

I need a Collector that's nearly identical to Collectors.toSet(), but with a custom finisher. I'd love to be able to do something like:

myCollector = Collectors.toSet();
myCollector.setFinisher(myCustomFinisher);

and be done, but that doesn't seem possible. The only alternative I can see is it essentially recreate Collectors.toSet() using Collector.of(), which is not very DRY.

Is there a way to take an existing Collector and amend it as described above?

EDIT

A few of the answers have recommended something like this:

  Collector<T, ?, Set<T>> toSet = Collectors.toSet();
  return Collector.of(
     toSet.supplier(),
     toSet.accumulator(),
     toSet.combiner(),
     yourFinisher,
     toSet.characteristics());

However my custom finisher isn't actually returning a Set; it's using the accumulated set to return something else. The fact that it's doing that is putting me into Generics hell which I'm still rummaging through..