ChromeDriver times out instantly when using Task.Run() with await Task.WhenAll()

Question

I'm trying to navigate to pages in parallel with ChromeDriver but as soon as I add "await" in front of the Task.WhenAll() call the driver times out within 1-2 seconds, not even close to the configured 120s timeout. What could be the reason for this and how can I fix it?

Using Task.WhenAll(..).Wait(..) seems to work fine. Why is that?

    var pages = new List();
    for (var i = 0; i < driverCount; i++)
    {
        var driver = _drivers[i];
        var url = urls[urlIndex++];
        Console.WriteLine($"{category.ToUpper()}: {urlIndex} of {urlCount} => {url}");

        pages.Add(Task.Run(() =>
        {
            try
            {
                driver.Navigate().GoToUrl(url);

            }
            catch (Exception e)
            {
                Console.WriteLine("Navigate() EXCEPTION:" + e);
                QuitAllChromeDrivers();
                throw;
            }
            
            Console.WriteLine($"Parsing: {driver.Url} - {driver.Title}");
            
            // parsing here ...
        }));
    }

    await Task.WhenAll(pages);

    OpenQA.Selenium.WebDriverException: The HTTP request to the remote WebDriver server for URL http://localhost:52594/session/5c138e57c554ccfd59a45b243512c476/title timed out after 120 seconds.
     ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
     ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
     ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
       --- End of inner exception stack trace ---
       at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
       at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
       at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       --- End of inner exception stack trace ---
       at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
       at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
       at OpenQA.Selenium.Remote.HttpCommandExecutor.MakeHttpRequest(HttpRequestInfo requestInfo)
       at OpenQA.Selenium.Remote.HttpCommandExecutor.Execute(Command commandToExecute)
       --- End of inner exception stack trace ---
       at OpenQA.Selenium.Remote.HttpCommandExecutor.Execute(Command commandToExecute)
       at OpenQA.Selenium.Remote.DriverServiceCommandExecutor.Execute(Command commandToExecute)
       at OpenQA.Selenium.WebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
       at OpenQA.Selenium.WebDriver.get_Title()
       at UnityAssetStoreScraper.Program.<>c__DisplayClass9_1.b__0() in C:\Users\Mee\RiderProjects\Scraper\Scraper\Program.Content.cs:line 74
       at System.Threading.Tasks.Task.InnerInvoke()
       at System.Threading.Tasks.Task.<>c.<.cctor>b__277_0(Object obj)
       at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)

user_1244630 · Answer

I suspect you've written a console application?

public static async Task Main()

Consider this part of the error:

The I/O operation has been aborted because of either a thread exit or an application request..

awaiting will return execution to the main thread, which has no other work, and will thus end.

When you use Task.WhenAll(pages).Wait(), you force the main thread to continue running until all of your driver tasks have completed.

You can test this using:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;

public class Program
{
    public static async Task Main()
    {
        List tasks = new List();
        for(var i = 0; i < 10; i++)
        {
            var captureI = i;
            tasks.Add(Task.Run(() => Console.WriteLine(captureI)));
        }
        Task.WhenAll(tasks).Wait(); // vs await Task.WhenAll(tasks);
    }
}

Some solutions are discussed here.

ChromeDriver times out instantly when using Task.Run() with await Task.WhenAll()

1 Answers