Detecting honest web crawlers

CynicalTyler_9304 0
Viewed 20022

I would like to detect (on the server side) which requests are from bots. I don't care about malicious bots at this point, just the ones that are playing nice. I've seen a few approaches that mostly involve matching the user agent string against keywords like 'bot'. But that seems awkward, incomplete, and unmaintainable. So does anyone have any more solid approaches? If not, do you have any resources you use to keep up to date with all the friendly user agents?

If you're curious: I'm not trying to do anything against any search engine policy. We have a section of the site where a user is randomly presented with one of several slightly different versions of a page. However if a web crawler is detected, we'd always give them the same version so that the index is consistent.

Also I'm using Java, but I would imagine the approach would be similar for any server-side technology.

bots web-crawler c#
7 Answers

You can find a very thorough database of data on known "good" web crawlers in the robotstxt.org Robots Database. Utilizing this data would be far more effective than just matching bot in the user-agent.

user_136750

One suggestion is to create an empty anchor on your page that only a bot would follow. Normal users wouldn't see the link, leaving spiders and bots to follow. For example, an empty anchor tag that points to a subfolder would record a get request in your logs...

<a href="dontfollowme.aspx"></a>

Many people use this method while running a HoneyPot to catch malicious bots that aren't following the robots.txt file. I use the empty anchor method in an ASP.NET honeypot solution I wrote to trap and block those creepy crawlers...

Dscoduc_519490

Any visitor whose entry page is /robots.txt is probably a bot.

user_136750 
void CheckBrowserCaps()
    {
        String labelText = "";
        System.Web.HttpBrowserCapabilities myBrowserCaps = Request.Browser;
        if (((System.Web.Configuration.HttpCapabilitiesBase)myBrowserCaps).Crawler)
        {
            labelText = "Browser is a search engine.";
        }
        else
        {
            labelText = "Browser is not a search engine.";
        }

        Label1.Text = labelText;
    }

HttpCapabilitiesBase.Crawler Property

user_37943000
Related