Asp.net core caching jwk during jwt validation

I'm relativly new to .NET. I am using ASP.NET Core 2.1. I am handing authentication using JWTs. As part of the jwt validation I call out to the issuer and get their public key to check my tokens integritry. So I do:

services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(o =>
    {
        o.Authority = Configuration["Jwt:Authority"];
        o.TokenValidationParameters = new TokenValidationParameters
        {
            ValidateIssuer = true,
            ValidateAudience = true,
            ValidateIssuerSigningKey = true,
            ValidIssuer = Configuration["Jwt:Issuer"],
            ValidAudience = Configuration["Jwt:Audience"],
            NameClaimType = ClaimTypes.NameIdentifier,
        };
    });

This does what I expect however it seems to go out to the issuing server for every request. What I would like is to cache the jwk I get back and only try to hit the authority if the validation fails(this could be because of key rotation, for example). What is strange is I'm not seeing an obvious way to define a caching policy in the configuration. I feel like this is a pretty standard use case and I might be missing something but I'm not having any luck looking through the docs. Could anyone suggest an approach for me here? Would I have to roll my own IConfigurationManager to enforce this? Any advice would be appreciated. Thanks much!