NET5 JWT Bearer Authentication not recognizing SSL certificate

Question

I have a .NET 5 WebApi using Grpc and an IdentityServer4 running behind a YARP reverse proxy. The reverse proxy is using a valid Let's Encrypt certificate and is routing requests to the other two which are listening on localhost:port and using a self signed certificate. They are running on Linux Mint 20.1 and I created the self signed certificate with OpenSSL and added it to /usr/local/share/ca-certificates/extra and ran update-ca-certificates to update the certificate store.

Everything runs fine, YARP recognizes the sefl signed certificate for routing the request but requests to the WebApi that require authorization throw this exception:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HMCHFQCVF5UE", Request id "0HMCHFQCVF5UE:0000000B": An unhandled exception was thrown by the application.
      System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'System.String'.
       ---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'System.String'.
       ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
       ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: NotTimeValid
         at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
         at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
         at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
         at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
         --- End of inner exception stack trace ---
         at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
         at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfigurationRetriever.GetAsync(String address, IDocumentRetriever retriever, CancellationToken cancel)
         at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
         --- End of inner exception stack trace ---
         at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
         at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
         at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
         at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.AuthenticateAsync()
         at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at Grpc.AspNetCore.Web.Internal.GrpcWebMiddleware.HandleGrpcWebRequest(HttpContext httpContext, ServerGrpcWebMode mode)
         at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
         at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

I'm using Microsoft.AspNetCore.Authentication.JwtBear 5.0.11 and that's where it seems to be coming from. The error would seem to suggest there is something wrong with the time on the cerfiticate but checking with OpenSSL both the not before and not after dates are fine. This is how it's setup in startup:

            services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
                .AddJwtBearer(options =>
                {
                    options.Authority = appSettings.Authorization.Authority;
                    options.TokenValidationParameters.ValidTypes = new[] { "at+jwt" };
                    options.TokenValidationParameters.ValidateAudience = false;
                });

I tried using both "https://localhost:port" and "https://real.hostname" for the Authority and the results were the same. I also tried using options.RequireHttpsMetadata = false; but it had no effect and I wouldn't really want to switch to normal http either.

I also tried running the same setup on my Windows machine using the same certificates and everything is working fine there. What could cause the different behavior and how could I get more information about what certificate it's actually trying to validate and how it determined the error?

user_7183682 · Answer

I managed to track down the cause and fix it.

The Cause

Microsoft.AspNetCore.Authentication.JwtBearer actually makes not 1 but 2 calls to IdentityServer4: one to /.well-known/openid-configuration to get the configuration and then a call to the endpoint returned in jwks_uriof the previous response. The first call was to a localhost:port endpoint using the self signed certificate and working normally but the 2nd was to a realdomain endpoint using the Let's Encrypt certificate and failing with the error in the OP.
The Let's Encrypt certificate had an expired certificate in the chain: it was using DST Root CA X3 instead of ISRG Root X1 (more info here: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/)

Fix for calls using different domains

Option 1 - fixing it in IdentityServer4

This can be achieved by changing the IdentityServer4 origin in Startup.cs:

        app.Use(async (ctx, next) =>
        {
            ctx.SetIdentityServerOrigin("origin");
            await next();
        });

Option 2 - fixing it in the API jwt authorization configuration

This requires implementing your own configuration manager:

public class CustomConfigurationManager : IConfigurationManager
{
    private readonly string authority;
    private readonly string authorityReturnedOrigin;

    public CustomConfigurationManager(string authority, string authorityReturnedOrigin)
    {
        this.authority = authority;
        this.authorityReturnedOrigin = authorityReturnedOrigin;
    }
    public async Task GetConfigurationAsync(CancellationToken cancel)
    {
        var httpClient = new HttpClient();

        var request = new HttpRequestMessage
        {
            RequestUri = new Uri($"{authority}/.well-known/openid-configuration"),
            Method = HttpMethod.Get
        };

        var configurationResult = await httpClient.SendAsync(request, cancel);
        var resultContent = await configurationResult.Content.ReadAsStringAsync(cancel);
        if (configurationResult.IsSuccessStatusCode)
        {
            var config = OpenIdConnectConfiguration.Create(resultContent);
            var jwks = config.JwksUri.Replace(authorityReturnedOrigin, authority);
            var keyRequest = new HttpRequestMessage
            {
                RequestUri = new Uri(jwks),
                Method = HttpMethod.Get
            };
            var keysResposne = await httpClient.SendAsync(keyRequest, cancel);
            var keysResultContent = await keysResposne.Content.ReadAsStringAsync(cancel);
            if (keysResposne.IsSuccessStatusCode)
            {
                config.JsonWebKeySet = new JsonWebKeySet(keysResultContent);
                var signingKeys = config.JsonWebKeySet.GetSigningKeys();
                foreach (var key in signingKeys)
                {
                    config.SigningKeys.Add(key);
                }
            }
            else
            {
                throw new Exception($"Failed to get jwks: {keysResposne.StatusCode}: {keysResultContent}");
            }

            return config;
        }
        else
        {
            throw new Exception($"Failed to get configuration: {configurationResult.StatusCode}: {resultContent}");
        }
    }

    public void RequestRefresh()
    {
        // if you are caching the configuration this is probably where you should invalidate it
    }
}

And then replacing the default configuration manager in Startup.cs:

services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
        .AddJwtBearer(options =>
        {
            options.Authority = appSettings.Authorization.Authority;
            options.TokenValidationParameters.ValidTypes = new[] { "at+jwt" };
            options.TokenValidationParameters.ValidateAudience = false;
            options.ConfigurationManager = new CustomConfigurationManager("authority", "authorityReturnedOrigin");
        }

Fix for the expired certificate chain

This is what worked for Linux Mint 20.1, other distros might have different way of handling things. Backing things up is also recommended.

Make sure openssl version is above 1.0.1 or 1.0.2 as those supposedly have issues (run openssl version to check)
Run ls -l | grep ISRG and delete what you find - this step might not be necessary but it's what I did
Go to /etc/ssl/certs
Run ls -l | grep DST and delete what you find
Download these files: isrgrootx1 isrgrootx2 letsencrytptr3
Copy them into /etc/ssl/certs Note that running cert update commands like update-ca-certificate might revert this.

There might be a better solution for this, maybe just updating to a newer OS version would've fixed it. This problem only apperead for http calls made to the API from code running on Linux Mint 20.1, on Windows things worked normally and accessing through the browser did not dispaly any SSL certificate errors.