User agent string with HTML::TreeBuilder

Viewed 77

I am using HTML::TreeBuilder->new_from_url($url), but the server blocks me due to incompatible user agent string. How can I change the user agent string with HTML::TreeBuilder to use new_from_url($url) directly, without a workaround of getting web content by the traditional way?

1 Answers

There is no way to set that in new_from_url(). You'll have to load LWP::UserAgent yourself and set the user agent string. Then you can parse the content with new_from_content().

However, here's a direct link to the new_from_url code. You can use that as the basis of your own new() method so that your program flow will read the same way.

So something like:

sub HTML::TreeBuilder::my_new_from_url {
  # copy and augment new_from_url
  ...
}

my $tree = HTML::TreeBuilder->my_new_from_url($url, { useragent => "foo" });

Now all of that right-way-to-do-it notwithstanding, you can play a dirty trick on LWP::UserAgent by overriding its private _agent method that stores the default user agent string. But be aware that you have voided the warranty and all broken module behavior is on you.

#the original 
sub _agent { "libwww-perl/$VERSION" }

Define in your code:

use LWP::UserAgent ();
sub LWP::UserAgent::_agent { "Mozilla forever" }

Or, more deviously:

use LWP::UserAgent ();

my $tree = do {
  local *LWP::UserAgent::_agent = sub { "Mozilla forever" };
  HTML::TreeBuilder->new_from_url($url);

};

Related