Spring Security - Prevent login if missing role

I have a spring application which uses LDAP authentication to authenticate users, combine this with a custom authority populator I can login to my application correctly. However I want to prevent users from logging in entirely if they don't have a specific role.

I've attempted to create a login success handler which will get the roles from the Authentication object and setAuthenticated(false) if they don't meet the requirements but this appears to be too late and still allows the user to proceed.

I could create a custom authentication provider but due to the LdapAuthenticationProviderConfigurer being so useful I'm not wanting to do that if possible.

I currently have the security configured with this:

auth.ldapAuthentication()
        .userSearchBase(ldapBase)
        .userSearchFilter(ldapFilter)
        .ldapAuthoritiesPopulator(new AuthorityPopulator(client))
        .contextSource(contextSource());

And my success handler:

protected void handle(HttpServletRequest request, HttpServletResponse response, Authentication authentication) throws IOException {
    String targetUrl = determineTargetUrl(request, response);
    List<String> roles = authentication.getAuthorities().stream().map(GrantedAuthority::getAuthority).collect(Collectors.toList());
    if(!roles.contains("admin")) {
        authentication.setAuthenticated(false);
    }

    redirectStrategy.sendRedirect(request, response, targetUrl);
}