Multiple Authentication Schemes and policies with WebAPI and NetCore 2.1

I am having a confusing time trying to get something to work that seems like it should be possible, but maybe I am misunderstanding something basic.

Our service written with NetCore 2.1 supports configurable authentication. It supports both ApiKey and Bearer Token authentication. One, Both, or neither of these options are to be allowed.

The bearer token authentication also has a policy involved, requiring a specific scope.

The ApiKey has no scope requirements.

What I've found given the way I set things up now is that the scoping policy tied to the Bearer authentication is being enforced on the ApiKey scheme as well, causing it to fail to authorize, despite authenticating properly in the ApiKey Authentication Handler.

The policies and authorization are added like so:

services.AddAuthorization(options =>
                         {
                           options.AddPolicy("BearerPolicy", builder =>
                           {
                                builder.RequireScope("Scope");
                                builder.AddAuthenticationSchemes("Bearer"); // shouldn't this make the policy only apply to Bearer Auth?
                           });
                           options.AddPolicy("ApiKeyPolicy", builder =>
                           {
                                builder.RequireClaim(ClaimTypes.Name,"Default");
                                builder.AddAuthenticationSchemes("ApiKey"); // shouldn't this make the policy only apply to ApiKey Auth?
                           });        
                         });

The ApiKeyPolicy was added as an attempt to solve the problem. I don't feel like the policy needs to be there for it. It seems like the "AddAuthenticationSchemes" property doesn't make the policy exclusively apply to that Authentication Type only, like you would think.

Controllers that need to be authenticated/authorized contain the following tags:

[Authorize(AuthenticationSchemes = "ApiKey", Policy = "ApiKeyPolicy")]
[Authorize(AuthenticationSchemes = "Bearer", Policy = "BearerPolicy")]

I would assume this setup should allow ApiKey and Bearer authentication, applying the correct policy depending which is being used.

However, it seems like ALL of the policies are considered, regardless of which Authentication is taking place.

Is this by design, or have I left something out? I stepped down into the authorization code of NetCore and found that it always has all of the policies, which I don't think is right, given the fact you can specify authentication schemes both in the policy, and the Authorize attribute tag.

I can provide more information as well. I think I've given the parts that would be needed.

We are using IdentityServer4 for the bearer token stuff.

Thank you!