Access denied for user with authority in Spring Cloud Security

I am building a user authentication system using spring cloud oauth-2 and spring cloud security. I want to have many authorities that can be assigned to user groups. The authorities are stored as enum constants as below:

public enum Authority {
    READ_USER(1, "View users and their details", "READ_USER"),
    WRITE_USER(2, "Create, Edit and Delete users and their details", "WRITE_USER"),
    // More authorities will be added
    ;

    Authority(int id, String description, String name) {
        this.id = id;
        this.description = description;
        this.name = name;
    }

    private final int id;
    private final String description;
    private final String name;

    public int getId() {
        return id;
    }

    public String getDescription() {
        return description;
    }

    public String getName() {
        return name;
    }
}

I have got my authentication server set up to issue JWT tokens to in-memory clients. All that works fine (I think).

Below is my UserDetails implementation

public class AuthPrincipal implements UserDetails {
    private String email;
    private String password;
    private Collection<? extends GrantedAuthority> authoritySet;

    public AuthPrincipal(User user) {
        this.email = user.getEmail();
        this.password = user.getPassword();
        this.authoritySet = user.getGroup().getAuthorities().stream().map(authority -> new SimpleGrantedAuthority(authority.getName())).collect(Collectors.toList());
    }

// Necessary getters

}

Like I said, the Authentication server is issuing tokens. The issue I am facing is with authorization. Below is my resource server http configuration:

@Configuration
@EnableWebSecurity()
@EnableGlobalMethodSecurity(prePostEnabled = true, securedEnabled = true)
public class ResourceServerConfig extends ResourceServerConfigurerAdapter {


    @Override
    public void configure(HttpSecurity http) throws Exception{
        http
                .authorizeRequests()
                .antMatchers("/").permitAll()


                // Users paths
                .antMatchers(HttpMethod.GET, "/users/**").hasAuthority(Authority.READ_USER.getName())
                .antMatchers(HttpMethod.POST, "/users/**").hasAuthority(Authority.WRITE_USER.getName())
                .antMatchers(HttpMethod.PUT, "/users/**").hasAuthority(Authority.WRITE_USER.getName())
                .antMatchers(HttpMethod.DELETE, "/users/**").hasAuthority(Authority.WRITE_USER.getName())
                .anyRequest().authenticated();
    }


}

After a successful authentication, I am getting a 403 access denied error for user with the required authority as shown in the security debug logs below:

2021-11-29 14:34:42.567 DEBUG 9832 --- [nio-8080-exec-3] o.s.security.web.FilterChainProxy        : Securing GET /users
2021-11-29 14:34:42.567 DEBUG 9832 --- [nio-8080-exec-3] s.s.w.c.SecurityContextPersistenceFilter : Set SecurityContextHolder to empty SecurityContext
2021-11-29 14:34:42.591 DEBUG 9832 --- [nio-8080-exec-3] p.a.OAuth2AuthenticationProcessingFilter : Authentication success: OAuth2Authentication [Principal=superuser@email.com, Credentials=[PROTECTED], Authenticated=true, Details=remoteAddress=0:0:0:0:0:0:0:1, tokenType=BearertokenValue=<TOKEN>, Granted Authorities=[{authority=READ_USER}, {authority=WRITE_USER}]]
2021-11-29 14:34:42.595 DEBUG 9832 --- [nio-8080-exec-3] o.s.s.w.a.i.FilterSecurityInterceptor    : Failed to authorize filter invocation [GET /users] with attributes [#oauth2.throwOnError(hasAuthority('READ_USER'))]
2021-11-29 14:34:42.604 DEBUG 9832 --- [nio-8080-exec-3] s.s.o.p.e.DefaultOAuth2ExceptionRenderer : Written [error="access_denied", error_description="Access is denied"] as "application/json" using [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@4cd51cd5]
2021-11-29 14:34:42.604 DEBUG 9832 --- [nio-8080-exec-3] s.s.w.c.SecurityContextPersistenceFilter : Cleared SecurityContextHolder to complete request

As it can be seen from the logs, the user has an authority of "READ_USERS" but the GET request was still denied. What am I missing?

@Entity
public class User extends BaseEntity{
    @Id
    private UUID id = UUID.randomUUID();
    @NotNull(message = "First Name cannot be blank")
    private String firstName;
    @NotNull(message = "Last Name cannot be blank")
    private String lastName;
    private String middleName;
    @NotNull(message = "Email cannot be blank")
    @Column(unique = true)
    private String email;
    private String phoneNumber;
    @JsonIgnore
    private String password;
    private String imageUrl;
    @ManyToOne
    private UserGroup group;

// Constructors, Getters and Setters
}

And UserGroup

@Entity
public class UserGroup extends BaseEntity{
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    private Long id;
    private String name;
    @ElementCollection(fetch = FetchType.EAGER)
    @Enumerated(EnumType.STRING)
    private Set<Authority> authorities = new HashSet<>();

// Getters and setters
}