I am testing a large amount of public proxies for validity. Optimal throughput is a necessity due to the size of this collection. This is simple enough to achieve using HttpWebRequest/Task, however, it results in many exceptions being thrown due to the fact that many of the proxies are invalid. These caught exceptions are causing terrible performance in my application. From my research, I've seen that exceptions have significant performance overhead.
I thought maybe the Socket class would be a good solution, but quickly found that it also throws an exception when it fails to connect to the remote endpoint. This makes sense, but in my use case, I do not want a failure to connect to result in an exception, similar to how various methods such as TryParse exist in the .NET framework. As far as I've been able to research, a similar method does not exist for what I am trying to achieve.
I would like to perform a direct system call to perform a web request using C#. I feel that I am going to need to go to a lower level of abstraction to achieve the behavior I've described, but I am open to and grateful for anyone's input. If I am correct that system calls will be needed, any example code would be extremely beneficial.
Current implementation:
private async Task Validate()
{
var proxies = new List<string>(); //In actual use, this contains 10,000+ proxies
var tasks = new List<Task>();
foreach (var proxy in proxies)
{
tasks.Add(IsValid(proxy));
}
await Task.WhenAll(tasks);
}
private async Task<bool> IsValid(string proxy)
{
try
{
var req = HttpWebRequest.Create("http://example.com");
req.Proxy = new WebProxy(proxy);
var resp = await req.GetResponseAsync();
using (var stream = resp.GetResponseStream())
{
using (var reader = new StreamReader(stream))
{
var html = await reader.ReadToEndAsync();
if (html.Contains("<title>Example Domain</title>"))
{
return true;
}
else
{
return false;
}
}
}
}
catch (Exception)
{
return false;
}
}