IDX10501: Signature validation failed. Key tried: 'System.IdentityModel.Tokens.X509AsymmetricSecurityKey'

I am trying to validate a token from Azure. I used Adal.js to get the token. When I try to validate the token I always get the same error message every time:

IDX10501: Signature validation failed. Key tried: 'System.IdentityModel.Tokens.X509AsymmetricSecurityKey'. token: '{"typ":"JWT",...

The token omitted from the message looks like what I can see on the client, and the information from the following 3 urls seems to be added into the data structures correctly, i.e. where I can see fields populated it is what I expect from looking at the links below and my token on the client.

https://login.windows.net/{id}.onmicrosoft.com/federationmetadata/2007-06/federationmetadata.xml

https://login.microsoftonline.com/{id}.onmicrosoft.com/.well-known/openid-configuration

https://login.microsoftonline.com/common/discovery/keys

But whenever I reach the final line ClaimsPrincipal claimsPrincipal = tokenHandler.ValidateToken(... I always get the same error.

Any ideas one how to make the token validate?

        // Get the jwt bearer token from the authorization header
        string jwtToken = null;
        AuthenticationHeaderValue authHeader = request.Headers.Authorization;
        if (authHeader != null)
        {
            jwtToken = authHeader.Parameter;
        }

        string issuer;
        List<SecurityToken> signingTokens;

        // The issuer and signingTokens are cached for 24 hours. They are updated if any of the conditions in the if condition is true.            
        if (DateTime.UtcNow.Subtract(_stsMetadataRetrievalTime).TotalHours > 24 || string.IsNullOrEmpty(_issuer) || _signingTokens == null)
        {
            // Get tenant information that's used to validate incoming jwt tokens
            string stsDiscoveryEndpoint = string.Format("{0}/.well-known/openid-configuration", authority);
            ConfigurationManager<OpenIdConnectConfiguration> configManager = new ConfigurationManager<OpenIdConnectConfiguration>(stsDiscoveryEndpoint);
            OpenIdConnectConfiguration config = await configManager.GetConfigurationAsync();
            _issuer = config.Issuer;
            _signingTokens = config.SigningTokens.ToList();

            _stsMetadataRetrievalTime = DateTime.UtcNow;
        }

        issuer = _issuer;
        signingTokens = _signingTokens;

        JwtSecurityTokenHandler tokenHandler = new JwtSecurityTokenHandler();

        TokenValidationParameters validationParameters = new TokenValidationParameters
        {
            ValidAudience = audience,
            ValidIssuer = issuer,
            IssuerSigningTokens = signingTokens,
            CertificateValidator = X509CertificateValidator.None
        };

        try {
            // Validate token.
            SecurityToken validatedToken = new JwtSecurityToken();
            ClaimsPrincipal claimsPrincipal = tokenHandler.ValidateToken(jwtToken, validationParameters, out validatedToken);
        }   

Update
Just incase there's something I'm missing when initializing the client and server.

Adal.js init options are:

    var endpoints = {
        "https://graph.windows.net": "https://graph.windows.net"
    };
    var configOptions = {
        tenant: "<ad>.onmicrosoft.com", // Optional by default, it sends common
        clientId: "<app ID from azure portal>",
        postLogoutRedirectUri: window.location.origin,
        endpoints: endpoints,
    }
    window.authContext = new AuthenticationContext(configOptions);

Server init options are:

    static string aadInstance = "https://login.microsoftonline.com/{0}";
    static string tenant = "<ad>.onmicrosoft.com";
    static string audience = "<app ID from azure portal>";
    string authority = String.Format(CultureInfo.InvariantCulture, aadInstance, tenant);

    static string scopeClaimType = "http://schemas.microsoft.com/identity/claims/scope";