Cannot access protected endpoint using Spring Security : Full authentication is required to access this resource

Using Spring 2.7 and Spring security, I implemented a custom Authentication manager, where users their email and an access token from Gitlab as a password.

Login does work, but I cannot access protected endpoints, here is the logs I get : It seems that the authentication context is set with the validated user : UsernamePasswordAuthenticationToken objet is shown. But later I get a Full authentication is required to access this resource, and the debug shows Principal=anonymousUser

: Found user in JWT : john.doe@example.com
: Found access_token in JWT : xxxxxxxxxxx
: Found user john.doe@example.com in cache
: User found using customUserDetailsService : john.doe
: User authorities found using customUserDetailsService : [ROLE_ADMIN, ROLE_USER]
: Authentication : 
: UsernamePasswordAuthenticationToken [Principal=com.example.juno.api.gitlab.model.CustomUserDetails@751e138e, Credentials=[PROTECTED], Authenticated=true, Details=WebAuthenticationDetails [RemoteIpAddress=0:0:0:0:0:0:0:1, SessionId=null], Granted Authorities=[com.example.juno.api.gitlab.model.CustomGrantedAuthority@68e3458c, com.example.juno.api.gitlab.model.CustomGrantedAuthority@44210794, com.example.juno.api.gitlab.model.CustomGrantedAuthority@10360f2a, com.example.juno.api.gitlab.model.CustomGrantedAuthority@1531ecc1, com.example.juno.api.gitlab.model.CustomGrantedAuthority@670c14ce]]
: Securing GET /auth/gitlab
: Set SecurityContextHolder to empty SecurityContext
: Set SecurityContextHolder to anonymous SecurityContext
: Request requested invalid session id 44BA45665B7EA4E693317CB8B22FD65C
: Failed to authorize filter invocation [GET /auth/gitlab] with attributes [authenticated]
: Full authentication is required to access this resource : /juno/auth/gitlab
: Cleared SecurityContextHolder to complete request
: Securing GET /error
: Set SecurityContextHolder to empty SecurityContext
: Set SecurityContextHolder to anonymous SecurityContext
: Secured GET /error
: filter invocation [/error] denied for AnonymousAuthenticationToken [Principal=anonymousUser, Credentials=[PROTECTED], Authenticated=true, Details=WebAuthenticationDetails [RemoteIpAddress=0:0:0:0:0:0:0:1, SessionId=null], Granted Authorities=[ROLE_ANONYMOUS]]

My security filters chain configuration is :

@Bean
public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
    
    http.csrf().disable();
    
    http.exceptionHandling().authenticationEntryPoint(restAuthenticationEntryPoint);
    
    http.sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS);
    
    http.authorizeRequests().antMatchers(AUTH_WHITELIST).permitAll();
    http.authorizeRequests().antMatchers(SWAGGER_WHITELIST).permitAll();
    http.authorizeRequests().antMatchers(String.format("/**/%s/**", JunoConstants.EXPOSED_ENDPOINT)).permitAll();
    
    http.authorizeRequests().anyRequest().authenticated();

    http.addFilterBefore(jwTokenAuthenticationFilter, UsernamePasswordAuthenticationFilter.class);
    
    http.authenticationManager(gitlabAuthenticationManager);

    return http.build();
}

And here is the part where I authenticate the user :

@Slf4j
@Component
@Order(Integer.MIN_VALUE)
public class JWTokenAuthenticationFilter extends OncePerRequestFilter {

    @Override
    public void doFilterInternal(
            HttpServletRequest request,
            HttpServletResponse response,
            FilterChain chain
    ) throws IOException, ServletException {

    ...
                        // Get user
                        UserDetails userDetails = customUserDetailsService.loadUserByEmailAndToken(email, accessToken);
                        
                        log.info("User found using customUserDetailsService : {}", userDetails.getUsername());
                        log.info("User authorities found using customUserDetailsService : {}", 
                                userDetails.getAuthorities().stream().map(GrantedAuthority::getAuthority)
                                        .collect(Collectors.toList()));
                    
                        // Create authentication
                        UsernamePasswordAuthenticationToken authentication = new UsernamePasswordAuthenticationToken(
                                userDetails, null,
                                userDetails.getAuthorities());

                        authentication.setDetails(
                                new WebAuthenticationDetailsSource().buildDetails(request)
                        );

                        SecurityContextHolder.getContext().setAuthentication(authentication);
                        log.info("Authentication : ");
                        log.info(SecurityContextHolder.getContext().getAuthentication().toString());
    ...
        chain.doFilter(request, response);
    }        
}