How to access session data in IAuthorizationHandler in ASP.NET Core 3.1?

I'm having difficulty accessing session data in a custom authorization handler, while it works elsewhere.

Session works properly in controller actions, but when I try to access it with IHttpContextAccessor.HttpContext.Session in a MyAuthorizationHandler.HandleRequirementAsync, I get a InvalidOperationException "Session has not been configured for this application or request.". The IHttpContextAccessor has access to query, cookies etc - but fails to access session data.

If I try to access the same MyAuthorizationHandler's IHttpContextAccessor from a controller, session data is available.

MyAuthorizationHandler is injected as a singleton in ConfigureServices, full order of initialization:

public void ConfigureServices(IServiceCollection services)
{
    services.AddDistributedMemoryCache(); 
    services.AddSession(); 
    services.AddDefaultIdentity...
    services.AddControllersWithViews();
    services.AddRazorPages();
    services.AddControllers...
    services.AddAuthentication...
    services.AddAuthorization...
    services.AddHttpContextAccessor(); 
    services.AddSingleton<IAuthorizationHandler, MyAuthorizationHandler>(); 
    services.Configure<CookiePolicyOptions>...
    services.AddMvc(o => o.EnableEndpointRouting = false);
}

public void Configure(...)
{
    app.UseRouting();
    app.UseAuthentication();
    app.UseAuthorization();
    app.UseSession();
    app.UseEndpoints...
    app.UseMvc();
}

Note: I cannot use claims in this handler, I need session data.