How do I give my API both App and Delegated permissions using Graph and Microsoft Identity Web

Problem Statement

I am writing an application that requires access to a OneDrive group that the user does not necessarily have access to. This (to me) means that I need to use application permissions, specifically "Sites.Selected". The application should however still be able to do things on behalf of the user.

I am trying to keep the application permissions as low as possible to keep the API more secure, so it would be great if I could use both Delegated and Application permissions at the same time.

What I've Done

According to what I read here, it seems like I would need two separate instances of the Graph Client and obtain a token for each of them. These posts are from 2018 though and things changed quite a bit since then. According to this it seems like it would be possible to have one instance of graph and using .WithAppOnly() and .WithScopes() you could tell Graph witch permissions to use.

I am currently using Microsoft.Identity.Web and .AddMicrosoftGraph with .WithAppOnly and .WithScopes but I am still unable to use the App Permissions. The .WithScopes functions are working but I get the below error when trying to use .WithAppOnly:

System.Collections.Generic.KeyNotFoundException: The given key 'Microsoft.Graph.AuthenticationHandlerOption' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Microsoft.Identity.Web.BaseRequestExtensions.SetParameter[T](T baseRequest, Action`1 action)

Azure App Permissions

SharePoint Permission Check Result

{
"@odata.context": "https://graph.microsoft.com/v1.0/$metadata#sites('SITE_ID')/permissions/$entity",
"id": "PERMISSION ID",
"roles": [
    "write"
],
"grantedToIdentities": [
    {
        "application": {
            "displayName": "MY API",
            "id": "API CLIENT ID"
        }
    }
]

}

Code

Startup.cs

services.AddMicrosoftIdentityWebApiAuthentication(Configuration, "AzureAd")
        .EnableTokenAcquisitionToCallDownstreamApi()
        .AddMicrosoftGraph(Configuration.GetSection("Graph"))
        //.AddMicrosoftGraphAppOnly(authenticationProvider => new GraphServiceClient(authenticationProvider))
        .AddInMemoryTokenCaches();

appsettings.json

  "AzureAd": {
    "Instance": "https://login.microsoftonline.com/",
    "ClientId": "{ClientId}",
    "TenantId": "{TenantId}",
    "CallbackPath": "/signin-oidc",
    "Audience": "{ClientId}",

    "ClientSecret": "{ClientSecret}",
    "ClientCertificates": [
    ]
  },

  "Graph": {
    "BaseUrl": "https://graph.microsoft.com/v1.0",
    "Scopes": "user.read",
    "DefaultScope": "https://graph.microsoft.com/.default"
  }

Graph Call works fine without the .WithAppOnly, but then it is still using the user to authenticate the Graph Client (If the user does not have access to the drive, the call fails)

    WorkbookSessionInfo res = await _graphServiceClient.Groups[_groupId].Drive.Items[productStructureCalculator.CalculatorId].Workbook
        .CreateSession(persistChanges)
        .Request()
        .WithAppOnly()
        .Header("Prefer", "respond-async")
        .PostAsync();