Identity Server 4 Client Works Even If Server Down

I have 2 separate services - some asp.net core service(let's call if FooService) and security service with identity server 4 installed (let's call it IdentityService)

On FooService I use Identity Server 4 AccessTokenValidation:

                .AddIdentityServerAuthentication("IdentityService",
                options =>
                {
                    options.JwtValidationClockSkew = TimeSpan.Zero;
                    options.Authority = $"http://localhost:5000";
                    options.RequireHttpsMetadata = false;
                    options.SupportedTokens = SupportedTokens.Jwt;
                });

And after it starts, if the IdentityServer is down, JWT could not be validated, because it doesnt find IdnetityServer4 endpoints on address options.Authority = $"http://localhost:5000 - its normal behavior.

But there could be a situation, when IdentityServer is still Up, it created JWT and FooService validated some of them - and then IdentityServer goes down - FooService still works with these JWT, that are not still expired. Also, it successfully validates valid tokens, but that first appeared in request. Its like, identity server 4 client caches auth scheme or smth.

I tried to play with some settings and parameters:

 options.SaveToken = false;
 options.BackChannelTimeouts = TimeSpan.FromSeconds(1);
 options.CacheDuration = TimeSpan.FromSeconds(1);
 options.EnableCaching = false;
 options.BackChannelTimeouts = TimeSpan.FromSeconds(1);
 options.DiscoveryDocumentRefreshInterval = TimeSpan.FromMinutes(5);

But it still works with IdentityServer down.

Is there any way to stop validating JWT's when IdentityServer is down? Like to force it validating tokens everytome using authority urls? Thanks!