Client address is not authorized and caller is not a trusted service in Azure

I'm working on Azure. I have a windows service which accesses the Azure Key Vault.

My code looks something like this:

public static async Task<string> GetToken(string authority, string resource, string scope)
{
    var authContext = new AuthenticationContext(authority);
    ClientCredential clientCred = new ClientCredential(...); //app id, app secret
    AuthenticationResult result = await authContext.AcquireTokenAsync(resource, clientCred);

    if (result == null)
        throw new InvalidOperationException("Failed to obtain the JWT token");

    return result.AccessToken;
}

public static string GetSecret(string secretName)
{
    KeyVaultClient keyVaultClient = new KeyVaultClient(GetToken);
    try
    {
        return keyVaultClient.GetSecretAsync("my-key-vault-url", secretName).Result.Value;
    }
    catch(Exception ex)
    {
        return "Error";
    }
}

After I build and deploy my windows service, I have started it. Then I'm getting this exception:

Client address (IPaddress) is not authorized and caller is not a trusted service

However, I am able to do a telnet to the key vault:

telnet projectName-keyvault 443

I have searched for this issue, but couldn't find any solution.Any help in this regard will be highly helpful.