NSwag + System.InvalidOperationException: The method 'get' on path '/Site' is registered multiple times

I just setup a new .NET Core 3.1 project, but I am having some issues when trying to use NSwag.AspNetCore, it has something to do with my controller but I can't figure it out. Here is my Site controller:

[Route("[controller]")]
[ApiController]
public class SiteController : BaseController
{
    private readonly ISiteService _siteService;
    public SiteController(ISiteService siteService, ILogger<SiteController> logger) : base(logger)
    {
        _siteService = siteService;
    }

    [HttpGet]
    public IActionResult GetAllSites() => (ReadService(() => _siteService.Sites()));

    [HttpGet("domainName/{domainName}")]
    public IActionResult GetByDomainName(string domainName) => (ReadService(() => _siteService.Site(domainName)));

    [HttpGet("status")]
    public Task<IActionResult> GetStatusAsync() => (ReadServiceAsync(async () => await _siteService.GetApiStatusAsync()));

    [HttpGet("lookupdata")]
    public Task<IActionResult> GetLookupDataAsync() => (ReadServiceAsync(async () => await _siteService.GetLookupDataAsync()));

}

I am hitting /Site and it works fine via postman, it calls GetAllSites().

BUT for some reason I am getting the following error when trying to generate the swagger json (hit the SwaggerUI):

System.InvalidOperationException: The method 'get' on path '/Site' is registered multiple times.
    at NSwag.Generation.AspNetCore.AspNetCoreOpenApiDocumentGenerator.AddOperationDescriptionsToDocument(OpenApiDocument document, Type controllerType, List`1 operations, OpenApiDocumentGenerator swaggerGenerator, OpenApiSchemaResolver schemaResolver)
    at NSwag.Generation.AspNetCore.AspNetCoreOpenApiDocumentGenerator.GenerateForControllers(OpenApiDocument document, IGrouping`2[] apiGroups, OpenApiSchemaResolver schemaResolver)
    at NSwag.Generation.AspNetCore.AspNetCoreOpenApiDocumentGenerator.GenerateAsync(ApiDescriptionGroupCollection apiDescriptionGroups)
    at NSwag.Generation.AspNetCore.AspNetCoreOpenApiDocumentGenerator.GenerateAsync(Object serviceProvider)
    at NSwag.AspNetCore.OpenApiDocumentProvider.GenerateAsync(String documentName)
    at NSwag.AspNetCore.Middlewares.OpenApiDocumentMiddleware.GenerateDocumentAsync(HttpContext context)
    at NSwag.AspNetCore.Middlewares.OpenApiDocumentMiddleware.GetDocumentAsync(HttpContext context)
    at NSwag.AspNetCore.Middlewares.OpenApiDocumentMiddleware.Invoke(HttpContext context)

I am not sure what I could be doing wrong here everything seems to be setup properly.

For what it's worth, I am also including the custom BaseController class if there is something wrong there:

[ApiController]
public abstract class BaseController : ControllerBase
{
    private ILogger _logger;
    protected BaseController(ILogger logger)
    {
        _logger = logger;
    }
}

If I were to decorate GetAllSites with:

[HttpGet("AllSites")]

It will work, but I don't want to do that, I want to be able to get all sites when I hit /site