c# .net 4.5 async / multithread?

Question

I'm writing a C# console application that scrapes data from web pages.

This application will go to about 8000 web pages and scrape data(same format of data on each page).

I have it working right now with no async methods and no multithreading.

However, I need it to be faster. It only uses about 3%-6% of the CPU, I think because it spends the time waiting to download the html.(WebClient.DownloadString(url))

This is the basic flow of my program

DataSet alldata;

foreach(var url in the8000urls)
{
    // ScrapeData downloads the html from the url with WebClient.DownloadString
    // and scrapes the data into several datatables which it returns as a dataset.
    DataSet dataForOnePage = ScrapeData(url);

    //merge each table in dataForOnePage into allData
}

// PushAllDataToSql(alldata);

Ive been trying to multi thread this but am not sure how to properly get started. I'm using .net 4.5 and my understanding is async and await in 4.5 are made to make this much easier to program but I'm still a little lost.

My idea was to just keep making new threads that are async for this line

DataSet dataForOnePage = ScrapeData(url);

and then as each one finishes, run

//merge each table in dataForOnePage into allData

Can anyone point me in the right direction on how to make that line async in .net 4.5 c# and then have my merge method run on complete?

Thank you.

Edit: Here is my ScrapeData method:

public static DataSet GetProperyData(CookieAwareWebClient webClient, string pageid)
{
    var dsPageData = new DataSet();

    // DOWNLOAD HTML FOR THE REO PAGE AND LOAD IT INTO AN HTMLDOCUMENT
    string url = @"https://domain.com?&id=" + pageid + @"restofurl";
    string html = webClient.DownloadString(url);
    var doc = new HtmlDocument();
    doc.LoadHtml(html );

    // A BUNCH OF PARSING WITH HTMLAGILITY AND STORING IN dsPageData 
    return dsPageData ;
}

c# .net 4.5 async / multithread?

4 Answers