Custom http security configuration along with OAuth2 resource server

I'm using Spring Security OAuth2 with a very basic configuration that is working fine. I now want to have a separate WebSecurityConfigurerAdapter that contains custom logic that determines whether someone has authority to access some endpoints. However, it is not executed no matter what I try. Below are my OAuth2 configurations and my findings on the topic. Authorization server:

@Configuration
@EnableAuthorizationServer
public class OAuth2AuthorizationServerConfig extends AuthorizationServerConfigurerAdapter {

    @Autowired
    private TokenStore tokenStore;

    @Autowired
    private AuthenticationManagerBuilder authenticationManager;

    @Override
    public void configure(AuthorizationServerEndpointsConfigurer endpoints)throws Exception {
        endpoints.authenticationManager(authentication -> authenticationManager.getOrBuild().authenticate(authentication)).tokenStore(tokenStore);
    }

    @Override
    public void configure(ClientDetailsServiceConfigurer clients) throws Exception {
        clients.inMemory().withClient("CLIENT_NAME")...;
    }

}

Resource server:

@Configuration
@EnableResourceServer
public class OAuth2ResourceServerConfig extends ResourceServerConfigurerAdapter {

    @Autowired
    private TokenStore tokenStore;

    @Override
    public void configure(HttpSecurity http) throws Exception {
        http.authorizeRequests().anyRequest().authenticated();
    }

    @Override
    public void configure(final ResourceServerSecurityConfigurer resources) throws Exception {
        resources.tokenStore(tokenStore);
    }
}

So far so good. When the custom WebSecurityConfigurerAdapter comes into action I start to have problems, though. Since the EnableResourceServer-annotated bean creates a WebSecurityConfigurerAdapter with Order(3) it gets executed on every request first, the user is successfully authenticated/authorized, but my custom logic in my WebSecurityConfiguration is not executed. On the other hand if i set WebSecurityConfiguration to have an Order(2) or less, the custom access rules are executed, but it always says they're coming from an anonymous user (since the rules in the bean created by @EnableResourceServer are not executed).

@EnableWebSecurity
@Configuration
public class WebSecurityConfiguration extends WebSecurityConfigurerAdapter {

    @Bean
    public TokenStore tokenStore() {
        return new InMemoryTokenStore();
    }

    @Override
    public void configure(WebSecurity web) throws Exception {
        web.ignoring().antMatchers("/...");
    }

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http.csrf().disable();

        http.authorizeRequests().antMatchers(HttpMethod.GET, "/path/**")
        .access("@security.hasPermission(authentication, 'SOME', 'VALUE')");

        http.authorizeRequests().anyRequest().authenticated();
    }
}

Just as a side note, the @security reference in the access rules is just a simple named Spring bean:

@Component("security")
public class SecurityService {
    public boolean hasPermission(Authentication authentication, String param, String anotherParam) { ... }
}

I have integration tests that validate the custom access rules in WebSecurityConfiguration and they work (since I skip the authentication there). I'd like to be able to use the resource server just for authentication and then my custom http security for authorization.