Using Custom Exception Handler with Vaadin23

I am trying to setup a global exception handler in my application. When I trigger a NullpointerException by clicking a button for example, only the DefaultErrorHandler interacts with it and prints the stacktrace to my console.

What am I doing wrong?

Sadly, none of the ways detailed in the documentation (Custom Error Handler or Custom Exception Handlers) seem to work for me.

@Tag(Tag.DIV)
public class CustomErrorHandler extends Component implements HasErrorParameter<NullPointerException> {

  @Override
  public int setErrorParameter(BeforeEnterEvent event,
      ErrorParameter<NullPointerException> parameter) {
    System.out.println("custom error handler caught NPE");
    return 0;
  }
}

public class CustomErrorHandler implements ErrorHandler {

  private static final Logger log = logger(CustomErrorHandler.class);

  @Override
  public void error(ErrorEvent errorEvent) {
    log.error("Something wrong happened", errorEvent.getThrowable());
    if(UI.getCurrent() != null) {
      UI.getCurrent().access(() -> {
        Notification.show("An internal error has occurred." +
            "Please contact support.");
      });
    }
  }
}

which is initialized in

@SpringComponent
public class ServiceListener implements VaadinServiceInitListener, SessionInitListener {

  @Override
  public void serviceInit(ServiceInitEvent event) {
    event.getSource().addSessionInitListener(this);
  }
  @Override
  public void sessionInit(SessionInitEvent event) throws ServiceException {
    event.getSession().setErrorHandler(new CustomErrorHandler());
  }
}

EDIT: I found out that it works if exceptions are caught and re-thrown.

button.addClickListener(it -> { 
  try {
    doSomeStuffProducingExceptions();
  } catch (Exception e) {
    throw e;
  }
});