Unable to authenticate using WSFederation after upgrading to .NET 5 from .NET Framework 4.8

I've recently converted my project from .NET Framework 4.8 to .NET 5. Everything is working except the ability for users to sign in when the authentication is passing through a reverse proxy.

When the users are connected to the VPN, everything works, but off the VPN, they get a 404 after signing into ADFS when trying to POST to /signin-wsfed.

I've added this to my Startup:

public void ConfigureServices(IServiceCollection services)
{
    services.Configure<ForwardedHeadersOptions>(options =>
    {
        options.KnownNetworks.Clear();
        options.KnownProxies.Clear();
        options.ForwardLimit = null;
        options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
    });
}

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    app.UseForwardedHeaders();
}

I've turned on some header debugging as prescribed in: https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?view=aspnetcore-5.0#forwarded-headers-middleware-options

I noticed that the X-Forwarded-For value when not using the UserForwardedHeaders is not equal to the X-Original-For when I have it turned on, the link suggests they should be the same.

NGINX is our reverse proxy, here is some of the config:

location / {
    proxy_pass https://redacted_ip_address/;
    proxy_redirect  off;
    proxy_http_version 1.1;
    proxy_set_header   Upgrade $http_upgrade;
    proxy_set_header   Connection keep-alive;
    proxy_set_header   Host $host;
    proxy_set_header   X-NginX-Proxy    true;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_set_header   X-Queue-Start "t=${msec}000";
    proxy_connect_timeout      90;
    proxy_send_timeout         90;
    proxy_read_timeout         90;
    proxy_buffer_size          4k;
    proxy_buffers              4 32k;
    proxy_busy_buffers_size    64k;
    proxy_temp_file_write_size 64k;
    proxy_cache_bypass $http_upgrade;
    client_max_body_size       50m;
    client_body_buffer_size    128k;
}

If anyone has any suggestions of something I could try, it would be greatly appreciated.