No authenticationScheme was specified, and there was no DefaultChallengeScheme found with default authentification and custom authorization

this worked for me

// using Microsoft.AspNetCore.Authentication.Cookies;
// using Microsoft.AspNetCore.Http;

services.AddAuthentication(CookieAuthenticationDefaults.AuthenticationScheme)
    .AddCookie(CookieAuthenticationDefaults.AuthenticationScheme,
        options =>
        {
            options.LoginPath = new PathString("/auth/login");
            options.AccessDeniedPath = new PathString("/auth/denied");
        });

Your initial statement in the marked solution isn't entirely true. While your new solution may accomplish your original goal, it is still possible to circumvent the original error while preserving your AuthorizationHandler logic--provided you have basic authentication scheme handlers in place, even if they are functionally skeletons.

Speaking broadly, Authentication Handlers and schemes are meant to establish + validate identity, which makes them required for Authorization Handlers/policies to function--as they run on the supposition that an identity has already been established.

ASP.NET Dev Haok summarizes this best best here: "Authentication today isn't aware of authorization at all, it only cares about producing a ClaimsPrincipal per scheme. Authorization has to be aware of authentication somewhat, so AuthenticationSchemes in the policy is a mechanism for you to associate the policy with schemes used to build the effective claims principal for authorization (or it just uses the default httpContext.User for the request, which does rely on DefaultAuthenticateScheme)." https://github.com/aspnet/Security/issues/1469

In my case, the solution I'm working on provided its own implicit concept of identity, so we had no need for authentication schemes/handlers--just header tokens for authorization. So until our identity concepts changes, our header token authorization handlers that enforce the policies can be tied to 1-to-1 scheme skeletons.

Tags on endpoints:

[Authorize(AuthenticationSchemes = "AuthenticatedUserSchemeName", Policy = "AuthorizedUserPolicyName")]

Startup.cs:

        services.AddAuthentication(options =>
        {
            options.DefaultAuthenticateScheme = "AuthenticatedUserSchemeName";
        }).AddScheme<ValidTokenAuthenticationSchemeOptions, ValidTokenAuthenticationHandler>("AuthenticatedUserSchemeName", _ => { });

        services.AddAuthorization(options =>
        {
            options.AddPolicy("AuthorizedUserPolicyName", policy =>
            {
                //policy.RequireClaim(ClaimTypes.Sid,"authToken");
                policy.AddAuthenticationSchemes("AuthenticatedUserSchemeName");
                policy.AddRequirements(new ValidTokenAuthorizationRequirement());
            });
            services.AddSingleton<IAuthorizationHandler, ValidTokenAuthorizationHandler>();

Both the empty authentication handler and authorization handler are called (similar in setup to OP's respective posts) but the authorization handler still enforces our authorization policies.

I landed on this question after I tried to use an AuthorizationHandler without authentication. I was trying to automatically verify a captcha based on a policy, so I could apply the policy to all the endpoints which had to be protected with a captcha.

I ended with the same failure when I wanted to throw 401 or 403 if the captcha couldn't be verified.

I don't like any of the solutions here because they are hacky, they try to use authentication and authorization when we are not really doing any, after all ASP.NET Core kind of ties Authorization with Authentication because in their design principle you can't decide if someone is authorized if you don't know who that person is.

This made me realize you can sometimes (depends on your requirements) use a resource filter instead of trying to hack the authorization filter to work as you want it to work, leaving the authentication/authorization system alone, and independent.

This solution might not be adapted to everyone but it might help those who like me do not need authentication but just decide if a controller should be accessible based on something else than the identity of the person using the endpoint.

public class YourResourceFilter : IAsyncResourceFilter
{
    //any services you might need through DI (dependency injection)
    private readonly ICaptchaService captchaService;
    public YourResourceFilter(ICaptchaService captchaService)
    {
        this.captchaService = captchaService;
    }
    //this is where you need to code any logic related to your requirements.
    public async Task OnResourceExecutionAsync(ResourceExecutingContext context, ResourceExecutionDelegate next)
    {
        //in my case I needed to get a header value, and verify it using my captcha service.
        //but you should be able to do almost anything here because you have access to DI.
        if (context.HttpContext.Request.Headers.TryGetValue("captcha", out var values))
        {
            var response = values.FirstOrDefault();
            if (response != null && await captchaService.VerifyCaptchAsync(response))
            {
                await next(); //if all is OK then keep going.
                return;
            }
        }
        //if not then set the status code to whatever you want. In my case I wanted to return 403 when the captcha couldn't be verified.
        context.HttpContext.Response.StatusCode = (int)HttpStatusCode.Forbidden;
    }
}

Now, how do we run this filter on the controllers? And how do we use dependency injection on the resource?

You'll need to create an attribute which inherits ServiceFilterAttribute as such:

/// <summary>
/// This applies your own resource filter whenever you apply this attribute on a controller endpoint.
/// </summary>
public class YourResourceFilterAttribute : ServiceFilterAttribute
{
    public YourResourceFilterAttribute() : base(typeof(YourResourceFilter))
    {
    }
}

Finally do not forget to register your filter with your dependency injection container:

builder.Services.AddSingleton<YourResourceFilter>();

And apply it to any controller which you want to apply this on:

[AllowAnonymous] //you can add this attribute is you want to bypass any authentication/authorization set up on that Asp.net core project, so you can use it no matter what configuration you have on the authentication/authorization system
[YourResourceFilter]
[HttpPost]
public Task ExampleAsync(CancellationToken cancellationToken = default)
{
    // whatever code your controller endpoint has
}

TL:DR:

Using a IAsyncResourceFilter and a ServiceFilterAttribute to apply your custom request logic is a more flexible and less hacky approach than the ones above.

1. Implement an IAsyncResourceFilter which holds your logic on your 'authorization' without authentication.
2. Inherit a ServiceFilterAttribute to create an attribute which applies your filter to whatever controller you put the attribute on. It's compatible with or without authentication. If you want to bypass any existing authentication/authorization system just apply the [AllowAnonymous] attribute, if you have any.
3. Register your filter created in 1. with your dependency injection container.
4. Add the attribute you created in 2. to any controller endpoints you want to apply your custom logic.

Conclusion

This approach is a lot more flexible than hacking the Authorization feature to work without authentication which it clearly wasn't designed to do. Plus you have the great advantage that this will work regardless of the authentication/authorization system, so if later you add any to your API, you can still verify these parameters on whatever endpoint you want. IMO this is a cleaner and simpler approach in the end.

Many answer above are correct but same time convoluted with other aspects of authN/authZ. What actually resolves the exception in question is this line:

services.AddScheme<YourAuthenticationOptions, YourAuthenticationHandler>(YourAuthenticationSchemeName, options =>
    {
        options.YourProperty = yourValue;
    })