.Net Core Authentication with JWT Asymmetric RSA Key fails - IDX10609: Decryption Failed. No Keys tried: token

I am working on an api that does authentication and returns some User Details as a response. The Authentication was using JWT Bearer Token and used a symmetric Key . Now the requirement has changed and I am expected to use a RSA Asymmetric Key.

I have changed the Startup.cs as follows:

ConfigureServices:

    services.AddAuthentication(options =>
            {
                options.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme;
                options.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme;

            })
                .AddJwtBearer(options =>
                {
                    options.SaveToken = true;
                    options.TokenValidationParameters = new TokenValidationParameters
                    {
                        ValidateAudience = false,
                        ValidateIssuer = false,
                        ValidateIssuerSigningKey = true,
                        IssuerSigningKey = GetRSASecurityPrivateKey(),
                        LifetimeValidator = LifetimeValidator
                    };
                })
            .AddOpenIdConnect(options =>
            {
                options.Events.OnAuthorizationCodeReceived = RedeemAuthorizationCodeAsync;
                oidcConfig.ConfigureOIDC(options);
            })
                .AddCookie(options => { options.ExpireTimeSpan = 5.Minutes(); });
            

GetRSASecurityPrivateKey method:

private RsaSecurityKey GetRSASecurityPrivateKey()
        {
            var privateJwk = new Dictionary<string, string>
            {
                { "p", "_A74Ay4AkmltkakQYOYWncHN-Lwbw0-1imbe0HyQubDXI2D9ubFgw9CY-lL6g4naSGAAjY8gPLsCd2HAnbvhzGynI3dGvnYKvIkj-UWFIysMFABYUjlf7BDX8SPopzS_RqzJCYQ0y7EP8eCSCpOdxPBywFg9HX0"},
                {"kty","RSA" },
                { "q", "knQngrwz8cR8JsT2fJZ5OwTVhOh7p7r79Zl3s_NdzK8yeHJXmi-YjfMasDnNSqN5dC5Yzo5Y3ZaB5OQzyy-xTQkdczbmk0bvixNYGArav5l3KYYsWdusjB9oRBh55VblsOwGNmV_b5lq31UXSqfaFGEtUX7kij_CKs"},
                {"d", "g18MuW7EfXCrNHl1ABsdr2FgvtGWYd7lxfBL1LW9vJyEnLTZkTZnOVSVmbHQkKCxG3fzhXVhYI47R4WMzQx9LEUZMA7pSjYOSoEwhSazyeTfJvEE9AcQh2dpvqbyHFk9YXFGgo2zbz_l04D5GykPEDrz5m5098juoaw_ekrk2BUrizZ4lApPwikExifr5NuzJw6V7yawrCYHwGPejjOhviDvSSXVLx9NrGYqN8vjNUSQJtMkcF6U1e3jj-Db7gh99EhSls-gg6VktORJedkKMyzNLqWDImIfuzjJaks66XNOf4FEDRnyYux13sWgK4e9Vlw48vxKYcGz9zoSQ"},
                { "e", "AQAB" },
                { "use", "enc"},
                { "qi", "9TLNzUsH9sOYcGpOhHkfny-CwN-ol3vFyDbEMjWlZWhdT-hYjh6_kL6HCAXL1JkR6H76lQI6bXrX3Z5kAgUSHJPJp3oHCDz17mgsF3lRLVqPgE7UJaQR2fLIb-xJ2Q_GPJLG6K6YT5-g1uBPsAIamVEf35Vm2jz-apZQ"},
                { "dp", "5tYClk7j4TJCURqm0EcHwVRDAJ-tQVJXbLfHKOJbXGOys0jZNN1YRBxebSnfGOX_ "},
                { "alg", "RS256"},
                { "dq", "CvxTimb7roMCnRXhN29CIL84FQQrYmWQSNf-Nq1mt9fcfVR2ZcZ0NxGpStbLI3InxiNN61_FEOI-IgtCgGsOAmTIuXLskdt4Rn20HYBiR2DecL7BlYsWrUp1bZeo5XZX8hER0eDpYIk"},
                { "n", "kDLnITJdvIs2dAimd0n-Wvry-AbPFpIV4Mr5zk3DHT5wsYHJIDtLk2Hc07DWh3j1-Pawf-I7OZIPbwauAqEbnSCEn-hUR8FkxTEectdx5wH4MMVW4tRpJMsNGqjjOwHZpZ-ic4TdxYcOpciO16s6q0HYHUCQwvxcFes3VYtwkJfMYlAwiDbjwal76poorEKTpNkpt73RJPqFHNWyYMfZyb2xZmy9q_zL7mJfWYZdQ2KX25wUtdTAPRqPdakr8eFbc-zAjxqnlxKLdpvwconG2tDfb-CXsCwUGafw"}
             };

    byte[] p = Base64Url.Decode(privateJwk["p"]);
    byte[] q = Base64Url.Decode(privateJwk["q"]);
    byte[] d = Base64Url.Decode(privateJwk["d"]);
    byte[] e = Base64Url.Decode(privateJwk["e"]);
    byte[] qi = Base64Url.Decode(privateJwk["qi"]);
    byte[] dq = Base64Url.Decode(privateJwk["dq"]);
    byte[] dp = Base64Url.Decode(privateJwk["dp"]);
    byte[] n = Base64Url.Decode(privateJwk["n"]);
    RSA keyPrivate = RSA.Create();
    RSAParameters keyParams = new RSAParameters();
    keyParams.P = p;
    keyParams.Q = q;
    keyParams.D = d;
    keyParams.Exponent = e;
    keyParams.InverseQ = qi;
    keyParams.DP = dp;
    keyParams.DQ = dq;
    keyParams.Modulus = n;
    keyPrivate.ImportParameters(keyParams);
    RsaSecurityKey issuerSigningKey = new RsaSecurityKey(keyPrivate);
    return issuerSigningKey;
}

I have changed the keys in the private key. When I run the application, I keep getting this error

I am not sure what is causing this error:

SecurityTokenDecryptionFailedException: IDX10609: Decryption failed. No Keys tried: token: '[PII is hidden. For more details, see https://aka.ms/IdentityModel/PII.]'

Any suggestions are welcome. This is a .NET core Project and I am not sure why the middleware cant find the token?