HostBuilder setup for testing API endpoints with xUnit

I am wondering if anyone can help me I have been struggling with a problem for many days and followed some examples but none of them seem to work. Basically I am looking for a way to make HTTP requests, example, a GET request to test the endpoint of an web API created with .net core 3.1 , I would of thought this was straightforward however I seem to be missing something, creating a simple test server.

[Fact]
public async Task ShouldReturnHelloWorld()
{
    var hostBuilder = new HostBuilder()
        .ConfigureWebHost(webHost =>
        {
            // Add TestServer
            webHost.UseTestServer();
            webHost.Configure(app => app.Run(async ctx => 
                await ctx.Response.WriteAsync("Hello World!")));
        });

    // Build and start the IHost
    var host = await hostBuilder.StartAsync();

    // Create an HttpClient to send requests to the TestServer
    var client = host.GetTestClient();

    var response = await client.GetAsync("/");

    response.EnsureSuccessStatusCode();
    var responseString = await response.Content.ReadAsStringAsync();
    Assert.Equal("Hello World!", responseString);
}

However, when I take this to the next step to create a test server for the API, I keep getting that the connection is refused or timed out. Now I now think this is normally for two reasons, the local host URL is wrong or something firewall related but this not the case. I checked my start file in the web API and there is nothing that be should stopping it running, sure if there was how could I run it normally. If anyone could help, please see my class below I am trying to create a test server for tests. It is sever related cause I cant get the server to run health checks like below, which do run the API normally.

    [Collection(TestCollectionDefinition.Api)]
    public class HostBuilderTest 
    {

        [Fact]
        public void IsApiHealthy()
        {
            var setup = new Infrastructure.InfrastructureSetup();
            var response = setup.HealthCheck();
            response.IsSuccessStatusCode.Should().BeTrue();
        }
}

And the test server:

[CollectionDefinition(Api)]
public class TestCollectionDefinition : ICollectionFixture<InfrastructureSetup>
{
    public const string Api = "Api";
}

public class InfrastructureSetup : IDisposable
{

    protected internal const string AutomationEnv = "Development";

    private static void SetEnvironmentVariable()
    {
        var env = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");

        Console.WriteLine($"ASPNETCORE_ENVIRONMENT > {env}");

        if (string.IsNullOrEmpty(env))
        {
            Console.WriteLine($"Setting env variable to {AutomationEnv}!");
            Environment.SetEnvironmentVariable("ASPNETCORE_ENVIRONMENT", AutomationEnv);
        }
    }

    private readonly CancellationTokenSource _cancel = new CancellationTokenSource();
    private IHost _host;
    public InfrastructureSetup()
    {
        SetEnvironmentVariable();

        Task.Factory.StartNew(() =>
        {
            //From the API (Program)
            var hostBuilder = Program.CreateHostBuilder(null);

            _host = hostBuilder.Build();

            try
            {
                _host.StartAsync(_cancel.Token);
                Task.Delay(Timeout.Infinite, _cancel.Token).Wait(_cancel.Token);
            }
            catch (OperationCanceledException)
            {
                Console.WriteLine("Operation cancelled!");
            }

        }, _cancel.Token, TaskCreationOptions.LongRunning, TaskScheduler.Current);

        Task.Delay(TimeSpan.FromSeconds(15)).Wait(); //wait for server to start
    }

    public void Dispose()
    {
        _host?.StopAsync(_cancel.Token)
            .Wait(TimeSpan.FromSeconds(15));
        _cancel?.Cancel();
        _cancel?.Dispose();
    }
}