Missing XML comments when generating swagger via Swashbuckle CLI tool

I am using Swashbuckle V4.0.0 with Asp.Net Core 2.2. I am using XML comments to document my Swagger. When I get the swagger via the HTTP endpoints it all works great. When I run dotnet swagger tofile I get the swagger but no XML comments included. Has anyone else had XML comments work via the CLI tool?

I have looked at the Swashbuckle CLI code and I am not sure how it calls my startup code when it create the builder. I have included my code for how I tell Swashbuckle which files to use. This code is in Startup.ConfigureServices. I have XML files in my main project and a couple of library projects so I wrote a function to go find them. This function returns valid files and works perfectly when requesting swagger from http. I am at loss for what I am doing wrong.

var xmlFiles = GetXmlFiles(apiName);
services.AddSwaggerGen(
    options =>
    {
        // add a custom operation filter which sets default values
        options.OperationFilter<SwaggerDefaultValues>();

        // integrate xml comments
        foreach (string s in xmlFiles)
        {
            options.IncludeXmlComments(s);
        }
    });