Get all links on the same domain using HtmlAgilityPack

user_1659806 0
Viewed 249

I'm writing code to crawl links on a given web page. I'm trying out HtmlAgilityPack to read the html contents (using www.google.co.uk in my example). The code I'm using is as follows:

class Program
{
    static async Task Main(string[] args)
    {
        var links = GetLinks(new Uri("https://www.google.co.uk"));
        foreach (var link in links)
        {
            Console.WriteLine(link);
        }
    }

    private static List<string> GetLinks(Uri uri)
    {
        var doc = new HtmlWeb().Load(uri);
        return doc.DocumentNode.Descendants("a")
            .Select(a => a.GetAttributeValue("href", null))
            .Distinct()
            .Where(u => !string.IsNullOrEmpty(u)).ToList();
    }
}

I'm stripping out nulls and duplicate links. This gives the following result:

 - https://www.google.co.uk/imghp?hl=en&tab=wi
 - https://maps.google.co.uk/maps?hl=en&tab=wl
 - https://play.google.com/?hl=en&tab=w8
 - https://www.youtube.com/?gl=GB&tab=w1 
 - https://news.google.com/?tab=wn
 - https://mail.google.com/mail/?tab=wm 
 - https://drive.google.com/?tab=wo
 - https://www.google.co.uk/intl/en/about/products?tab=wh
 - http://www.google.co.uk/history/optout?hl=en 
 - /preferences?hl=en
 - https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.co.uk/&ec=GAZAAQ
 - /advanced_search?hl=en-GB&amp;authuser=0 
 - /intl/en/ads/ 
 - /services/
 - /intl/en/about.html
 - https://www.google.co.uk/setprefdomain?prefdom=US&amp;sig=K_eDMDym3RsPb7-MzvJkS4b2Eg4ns%3D
 - /intl/en/policies/privacy/ 
 - /intl/en/policies/terms/

I'd like to narrow down the links even further to select links that only match the same subdomain "www.google.co.uk", including relative URL's. The resulting list would be narrowed down to this:

 - https://www.google.co.uk/imghp?hl=en&tab=wi
 - https://www.google.co.uk/intl/en/about/products?tab=wh
 - http://www.google.co.uk/history/optout?hl=en 
 - /preferences?hl=en
 - /advanced_search?hl=en-GB&amp;authuser=0 
 - /intl/en/ads/ 
 - /services/
 - /intl/en/about.html
 - https://www.google.co.uk/setprefdomain?prefdom=US&amp;sig=K_eDMDym3RsPb7-MzvJkS4b2Eg4ns%3D
 - /intl/en/policies/privacy/ 
 - /intl/en/policies/terms/

I'm looking to modify the code above in the most efficient way to achieve this but not sure of the best way to approach it with HtmlAgilityPack. I've arrived at this solution:

private static List<string> GetLinks(Uri uri)
{
    var doc = new HtmlWeb().Load(uri);
    return doc.DocumentNode.Descendants("a")
        .Select(a => 
        { 
            var val = a.GetAttributeValue("href", null);

            if (val.StartsWith("/"))
                val = $"{uri.Scheme}://{uri.Host}{val}";

            return val;
        })
        .Distinct()
        .Where(u =>
        {
            return !string.IsNullOrEmpty(u) 
                   && u.Contains(uri.Host); // using contains here is a problem
        }).ToList();
}

I'm very conscious of the amount of string manipulation involved here in changing relative urls to fully qualified and also matching on "Contains", which could leave incorrect results. Does anyone have a less wasteful (string comparison and manipulation) solution to this?

Any advice greatly appreciated!

web-crawler html-agility-pack dom c#
1 Answers

Unless your HTML is huge and has a multitude of links, I wouldn't be that worried about optimizing for performance here.

It sounds like the main problem is just finding a replacement for Contains in the Where clause. I would use a Regex for that. You can build the match pattern once, at the beginning of your method, and then use IsMatch inside the Where, like so:

private static List<string> GetLinks(Uri uri)
{
    var regex = new Regex("^http(s)?://" + uri.Host, RegexOptions.IgnoreCase);
    var doc = new HtmlWeb().Load(uri);

    return doc.DocumentNode
        .Descendants("a")
        .Select(a =>
        {
            var val = a.GetAttributeValue("href", string.Empty);
            return val.StartsWith("/") ? uri.GetLeftPart(UriPartial.Authority) + val : val;
        })
        .Distinct()
        .Where(u => !string.IsNullOrEmpty(u) && regex.IsMatch(u))
        .ToList();
}

Fiddle: https://dotnetfiddle.net/BVdv1Y

user_102630
Related