JSON Web Token validity verification passed on .NET Core but failed on .NET 4.6.1

I am trying to varify validity of JSON Web Token in both .NET Core and .NET 4.6.1 by using Microsoft.IdentityModel.Tokens library. I dug into source code and noticed that AsymmetricSignatureProvider differently handle signaure verification in method bool Verify(byte[] input, byte[] signature) for .NET Core and other versions, which is also place where my verification failed. You can find source code in this link.

In general I am first downloading JSON Web Key Sets, then I am building security keys and last I am building TokenValidationParameters and validating token.

My code starts here

Downloading JSON Web Key Sets:

string data = null;
using (WebClient client = new WebClient())
{
    data = client.DownloadString(URL_TO_JWKS);
}

var jwks = new JsonWebKeySet(data);

Here I am building security keys from JSON Web Key Sets. Please notice how I decoding exponent and modulus, maybe is wrong in .NET 4.1.6:

    var keys = new List<SecurityKey>();
    foreach (var webKey in jwks.Keys)
    {
         var e = Base64UrlEncoder.DecodeBytes(webKey.E);
         var n = Base64UrlEncoder.DecodeBytes(webKey.N);
         var key = new RsaSecurityKey(new RSAParameters { Exponent = e, Modulus = n })
          {
             KeyId = webKey.Kid
          };

          keys.Add(key);
   }

Here I am verifying token:

var token = new JwtSecurityToken(tokenString);
ClaimsPrincipal claimsPrincipal = null;

            var parameters = new TokenValidationParameters()
            {
                ValidAudiences = token.Audiences,
                IssuerSigningKeys = keys,
                NameClaimType = nameClaimType ?? JwtRegisteredClaimNames.Sub,
                ValidIssuers = new[] { token.Issuer }
            };

            bool isValid = false;

            try
            {
                string jwt = token.RawData;
                SecurityToken securityToken = null;
                claimsPrincipal = new JwtSecurityTokenHandler().ValidateToken(jwt, parameters, out securityToken);
                isValid = claimsPrincipal.Identity.IsAuthenticated;
            }
            catch (SecurityTokenExpiredException)
            {
            }
            catch (SecurityTokenInvalidSignatureException ex)
            {
            }

What is correct way to verify validity for both .NET 4.6.1 and .NET Core?