How to log out External Authentication Schemes like Facebook in ASP .NET MVC Core?

I have a ASP .NET MVC Core app and I am implementing a very simple login page where I use IdentityUser to let the user login with Username, Email and Password. I also am using External Authentication Providers such as Facebook as well. Normal login/logout is working as it should. And when it comes to facebook, the login works fine:

 public IActionResult FacebookLogin()
    {
        AuthenticationProperties authProps = new AuthenticationProperties
        {
            RedirectUri = Url.Action("Index", "Home")
        };

        return Challenge(authProps, "Facebook");
    }

But I have problem to logout when I am logged in using Facebook. Here is my logout code:

    [HttpPost]
    [ValidateAntiForgeryToken]
    public async Task<IActionResult> Logout()
    {
        await _signinManager.SignOutAsync();
        return RedirectToAction("Index", "Home");
    }

No matter what I do, I cant logout when I'm logged in using Facebook Any Help appreciated. It is worth noting that I have a partial view as follows:

Whenever I login with facebook it the IsAuthenticated flag is set to true, but when I log out it is never set to false