WS-Federation not working with .Net Core 3.1

We're migrating a ASP.net Framework (4.8) MVC app to .Net Core 3.1. The application is using WsFederation configured to use ADFS to route back the user to /account/ExternalLoginCallback. In the callback claims are fetched to populate a model and lastly redirected to /account/ExternalLoginConfirmation after which the user is created in our database (which we need)

Applying the same config to our migrated .Net Core app, is not working :(

public async Task<IActionResult> ExternalLoginCallback(string returnUrl = null, string remoteError = null)
{
    var result = await HttpContext.AuthenticateAsync(IdentityConstants.ExternalScheme);  // result is null
    var info = await signInManager.GetExternalLoginInfoAsync(); // info is also null

We configure the WSFed in the ConfigureServices as:

authServices.AddWsFederation(externalLoginProvider.Domain, options =>
{
    options.Wtrealm = externalLoginProvider.Realm;
    options.MetadataAddress = externalLoginProvider.MetadataAddress;
    options.Wreply = "/Account/ExternalLoginCallback/" + externalLoginProvider.ProviderName;
});

I have been trying to use the options.CallbackPath option, but that only madee things worse...

Thank you for the help