WebRequest's GetResponseAsync unstable while GetResponse isn't

Viewed 410

I'm performing many requests through a collection of URLs so I can check which ones could retrieve a PDF file.

In order to do this, I create a WebRequest with HEAD method and checking afterwards the response I get.

When I perform the webRequest.GetResponse() (sync) everything seems to work fine and every request is being "fired".

On the other hand, when I perform the await webRequest.GetResponseAsync() for the third time it simply doesn't reach the end of the method.

Since it's actually into a try-catch in order to handle possible unreachable hosts, it simply ignores that link.

Async version:

private async Task<IEnumerable<string>> GetLinksContainsDownloadablePdfAsync(IEnumerable<string> linksInMail)
{
    var downloadableLinks = new List<string>();
    foreach (var link in linksInMail)
    {
        var headRequest = WebRequest.Create(link);
        headRequest.Method = "HEAD";
        try
        {
            var responseTest = await headRequest.GetResponseAsync();
            if (responseTest.Headers["Content-Type"].Contains("application/pdf"))
                downloadableLinks.Add(link);
        }
        catch (WebException)
        {
            //If it's not accesible, just ignore it
        }
    }
    return downloadableLinks;
}

Sync version:

private IEnumerable<string> GetLinksContainsDownloadablePdf(IEnumerable<string> linksInMail)
{
    var downloadableLinks = new List<string>();
    foreach (var link in linksInMail)
    {
        var headRequest = WebRequest.Create(link);
        headRequest.Method = "HEAD";
        try
        {
            var responseTest = headRequest.GetResponse();
            if (responseTest.Headers["Content-Type"].Contains("application/pdf"))
                downloadableLinks.Add(link);
        }
        catch (WebException)
        {
            //If it's not accesible, just ignore it
        }
    }
    return downloadableLinks;
}

Could anyone help throwing out some light here?

I'm not getting any Exception actually and the debugger isn't even reaching the return downloadableLinks line.

Note this is going to run in a server, so I'm specially interested to be multi-thread friendly.

EDIT: This method is not the only one which is async and I'm calling it from other async methods, so I'm supposed to be correctly handling the Task itself.

This is the method where I'm calling the GetLinksContainsDownloadablePdfAsync method

protected override async Task<IEnumerable<StoredFile>> ExtractPdfAsync(EmailMessageItem message, DocumentInfo documentInfo)
{
    var document = new HtmlDocument();
    document.LoadHtml(message.Body.HtmlBody);
    var validLinks = await new LinkHelper(document).GetValidLinksInHtmlBodyAsync();

    var pdfFiles = await DownloadPdfFromLinksAsync(documentInfo, validLinks);
    return pdfFiles;
}

EDIT 2: In order to provide more information to perform tests, these are links found on e-mails, and the e-mail I'm having issues with is an e-mail from a daily online newspaper. I've debugged every single call and noticed it's producing 3 calls since it's diverted with some redirects (3xx). Here is a link to pastebin with almost every link (removed some due to privacy-subscription issues)

1 Answers

You're not properly disposing the response object. I was able to reproduce your issue with the debugger, and after a few hits it became unstable and didn't hit the next line.

I think you may be able to solve this by disposing the response before getting a new one:

using (var response = await headRequest.GetResponseAsync())
{
    if (response.Headers["Content-Type"].Contains("application/pdf"))
    {                            
        downloadableLinks.Add(link);
    }
}
Related