Blazor - App Secured using AAD but how do I make calls to my API?

I successfully used this article to Secure my WASM app and the default WeatherForecastController. What I want to do now is extend the RemoteUserAccount to add the ClientId so it can be added to the User's Claims and always accessible.

What I am just now realizing is that the WeatherForecastController authorizes without issue but if I make any other Controllers and try to call an endpoint, I always get 401.

When a user is Authenticated, I have attempted to the do the below, but it only works if I remove the [RequiredScope(RequiredScopesConfigurationKey = "AzureAd:Scopes")] from the controller.

 var initialUser = await base.CreateUserAsync(account, options);
        if (initialUser.Identity.IsAuthenticated)
        {
            var userIdentity = (ClaimsIdentity)initialUser.Identity;

            var httpClient = _httpClientFactory.CreateClient();
            httpClient.BaseAddress = new Uri("https://localhost:5001/");
            var httpResponseMessage = await httpClient.GetAsync($"AdUser?ADObjectId=xxxxx");
            if (httpResponseMessage.IsSuccessStatusCode)
            {
                using var contentStream = await httpResponseMessage.Content.ReadAsStreamAsync();

                var adUser = await JsonSerializer.DeserializeAsync<AduserModel>(contentStream);
            }

            foreach (var role in account.Roles)
            {
                userIdentity.AddClaim(new Claim("appRole", role));
            }
        }

I think I need to reuse the httpClient from my Program.cs but I've never done any of this before, I'm usually the UI dev.

builder.Services.AddHttpClient("BlazorWasmHostedAAD.ServerAPI", client => client.BaseAddress = new Uri(builder.HostEnvironment.BaseAddress))
.AddHttpMessageHandler<BaseAddressAuthorizationMessageHandler>();

// Supply HttpClient instances that include access tokens when making requests to the server project
  builder.Services.AddScoped(sp => sp.GetRequiredService<IHttpClientFactory>().CreateClient("BlazorWasmHostedAAD.ServerAPI"));

  builder.Services.AddMsalAuthentication<RemoteAuthenticationState, SecureUserAccount>(options =>
{
builder.Configuration.Bind("AzureAd", options.ProviderOptions.Authentication);
options.ProviderOptions.DefaultAccessTokenScopes.Add("api://xxx/BlazorHostedAPI.Access");
options.UserOptions.RoleClaim = "appRole";
})
 .AddAccountClaimsPrincipalFactory<RemoteAuthenticationState, SecureUserAccount, SecureAccountFactory>();