How to access KeyVault from an API running as a windows service

I am building a POC where I have two APIs.

1. ClientApi
2. ServiceApi

The ClientApi calls the ServiceApi. I leverage the IDownstreamWebApi interface to make it work.

Both APIs are registered in Azure AD. KeyVault is used to get a certificate for the ClientApi to get a token for the ServiceApi.

It works from Visual Studio, where I leveraged the Azure Service Authentication extension.

The code is very simple:

var response = await downstreamWebApi.CallWebApiForAppAsync("ServiceApi", "Bearer", options =>
            {
                options.HttpMethod = HttpMethod.Get;
                options.RelativePath = "data/items";
            });

The configuration looks like this:

"AzureAd": {
    "Instance": "https://login.microsoftonline.com/",
    "ClientId": "[ClientApi Id]",
    "TenantId": "[Tenant Id]",

    // To call an API
    "ClientCertificates": [
      {
        "SourceType": "KeyVault",
        "KeyVaultUrl": "[key vault url]",
        "KeyVaultCertificateName": "client-cert"
      }
    ]
  },
  "ServiceApi": {
    "BaseUrl": "https://localhost:44376",
    "Scopes": "api://[ServiceApi Id]/.default"
  },

Question: In production, both the APIs will run as windows services hosted on an on-premise server.

I need to configure the access to the KeyVault for the ClientApi.

Is there a way to do that and keep the IDownstreamWebApi?

Or do I need to rewrite the code using a confidential client and KeyVault client? E.g. like here https://github.com/Azure-Samples/active-directory-dotnetcore-daemon-v2/blob/master/3-Using-KeyVault/README.md.