C# TokenValidationParameters for JwtSecurityTokenHandler having no effect on validation?

Hi and apologies for having a semi-open question, but I'm using Microsoft.IdentityModel.Tokens.TokenValidationParameters to verify a Bearer token against some Siging Key, but some of the parameters don't seem to have any effect on the outcome.

I'd like to verify that the SigningKey is using a specific algorithm and also that the SigningKey itself is valid.

I'm basically using:

TokenValidationParameters validationParameters = new TokenValidationParameters
{
    ValidIssuer = ...,
    ValidateAudience = true,
    ValidAudience = ...,
    RequireSignedTokens = true,
    //ValidateIssuerSigningKey = true, //tried with this included and excluded, but no difference
    IssuerSigningKey = //Key from ... /.well-known/openid-configuration/jwks,
    //AlgorithmValidator = //no need to us this from what I understand since there should be a default one ? 
    ValidAlgorithms = new List<string>(){ "RS256" }
};

try {
    SecurityToken validatedToken;
    JwtSecurityTokenHandler handler = new JwtSecurityTokenHandler();
    var claimsPrincipal = handler.ValidateToken(token, validationParameters, out validatedToken);
}
catch....

but both the ValidAlgorithms and IssuerSigningKey don't seem to have any affect. I get the Key from the Token issuer, so I my general assumption is that it's valid, so I tried changing parts of it before using it in the TokenValidationParameters , but regardless of what I do - nothing changes. There's no Exception or error or anything, like the Key is just completely ignored.

Same for ValidAlgorithms. If I understand correctly then the default should be that any valid Algorithm is accepted, but since I'd like to make sure that the Key uses RS256, I only want to check for this 1, but regardless of what I enter there, no changes, like it's completely ignored.

The thing is that the other parameters work just fine and if any of them fail the verification, then I get an appropriate Exception, but for those 2 nothing happens .. :|