AuthorizationPolicy set status code

I have an AuthorizationHandler and I would like to return a custom statuscode ( or text ) right now I am just trying to make it return 429 ( which is for rate limit but that's not relevant )

public class ReCaptchaHandler : AuthorizationHandler<ReCaptchaRequirement>
{
    private readonly ReCaptcha _reCaptcha;
    private readonly IHttpContextAccessor _accessor;

    public ReCaptchaHandler(ReCaptcha reCaptcha, IHttpContextAccessor accessor)
    {
        _reCaptcha = reCaptcha;
        _accessor = accessor;
    }
    protected override  Task HandleRequirementAsync(AuthorizationHandlerContext context,
        ReCaptchaRequirement requirement)
    {

        context.Fail();
        _accessor.HttpContext.Response.StatusCode = 429;
        return Task.CompletedTask;
    }
}

public class ReCaptchaRequirement :  IAuthorizationRequirement
{
}

as you can see, I am trying to set the status = 429 with _accessor.HttpContext.Response.StatusCode = 429;

I have tried using the http context from var filterContext = context.Resource as AuthorizationFilterContext; and then filterContext.HttpContext

I create my policy as follows

services.AddSingleton<IAuthorizationHandler, ReCaptchaHandler>();
services.AddSingleton<IHttpContextAccessor, HttpContextAccessor>();
services.AddAuthorization(options =>
{
    options.AddPolicy("ReCaptcha",
        policy => policy.Requirements.Add(new ReCaptchaRequirement()));
});

The code runs and protects the route marked with [Authorize(Policy = "ReCaptcha")] but the status is always 400 unless I throw an exception.