Scraping specific pieces of data from website in C# using HtmlAgilityPack

user_10343353 0
Viewed 500

I am tyring to use HtmlAgilityPack to scrape a web page for a certain nested div class that contains a span tag with the data I want to extract

The full XPath to the element's text I want:

/html/body/div[2]/div/div[1]/div/table/tbody/tr/td/span

My code:

  static void Main(string[] args)
        {
            HtmlAgilityPack.HtmlWeb web = new HtmlAgilityPack.HtmlWeb();
            HtmlAgilityPack.HtmlDocument doc = web.Load("http://watchout4snakes.com/wo4snakes/Random/RandomParagraph");

            var paragraph = doc.DocumentNode.SelectNodes("//div[@class='mainBody']//div[@class='content']//div[@class='resultContainer']" +
                "//div[@class='resultBox']//table[@class='paragraphResult']").ToList();

            foreach (var item in paragraph)
            {
                Console.WriteLine(item.InnerText);
            }
        }

I've tried putting the full XPath into the doc.DocumentNode.SelectNodes() as well as just the Xpath which is //*[@id='result'] My issue is that it either returns nothing or I get an error saying Unhandled exception. System.ArgumentNullException: Value cannot be null. (Parameter 'source') on the doc.DocumentNode.SelectNodes() line.

web-scraping html-agility-pack html c#
1 Answers

The problem source is that Paragraph is loaded through AJAX, by JS script after the page is loaded. If you open Dev Tools/Network in the Browser, you can see that.

The loaded page part looks like this

<table class="paragraphResult">
    <tr>
        <td>
            <span id="result"></span>
        </td>
    </tr>
</table>

Nothing there inside.

And after load, page sends the AJAX POST request to the same URL and gets the plain string, with parameters

Subject1: "",
Subject2: ""

And how it looks in URL-Encoded format

Subject1=&Subject2=

To perform emulating the Form behavior, you may send the POST request manually. HtmlAgilityPack is useless here because the received data is simply a plain string.

class Program
{
    private static readonly HttpClient client = new HttpClient();

    static async Task Main(string[] args)
    {
        Console.Write("Proper Noun 1: ");
        string subject1 = Console.ReadLine();
        Console.Write("Proper Noun 2: ");
        string subject2 = Console.ReadLine();
        Dictionary<string, string> parameters = new Dictionary<string, string>()
        {
            { "Subject1", subject1 },
            { "Subject2", subject2 }
        };
        try
        {
            string result = await PostHTTPRequestAsync("http://watchout4snakes.com/wo4snakes/Random/RandomParagraph", parameters);
            Console.WriteLine(result);
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
        }
        Console.ReadKey();
    }

    private static async Task<string> PostHTTPRequestAsync(string url, Dictionary<string, string> data)
    {
        using (HttpContent formContent = new FormUrlEncodedContent(data))
        using (HttpResponseMessage response = await client.PostAsync(url, formContent).ConfigureAwait(false))
        {
            response.EnsureSuccessStatusCode();
            return await response.Content.ReadAsStringAsync().ConfigureAwait(false);
        }
    }
}

Console output

Proper Noun 1: test
Proper Noun 2: StackOverflow
When will the bass garage StackOverflow? A digest sighs test below the objective card. Why won't test finish behind a girlfriend? A heating science approaches test.

Works well. All you need to achieve that is properly analyse HTTP traffic in the browser. Also HttpClient is powerful tool to interact with web.

user_128880240
Related