I have a web page with a bunch of links. I want to write a script which would dump all the data contained in those links in a local file.
Has anybody done that with PHP? General guidelines and gotchas would suffice as an answer.
15 Answers
Meh. Don't parse HTML with regexes.
Here's a DOM version inspired by Tatu's:
<?php
function crawl_page($url, $depth = 5)
{
static $seen = array();
if (isset($seen[$url]) || $depth === 0) {
return;
}
$seen[$url] = true;
$dom = new DOMDocument('1.0');
@$dom->loadHTMLFile($url);
$anchors = $dom->getElementsByTagName('a');
foreach ($anchors as $element) {
$href = $element->getAttribute('href');
if (0 !== strpos($href, 'http')) {
$path = '/' . ltrim($href, '/');
if (extension_loaded('http')) {
$href = http_build_url($url, array('path' => $path));
} else {
$parts = parse_url($url);
$href = $parts['scheme'] . '://';
if (isset($parts['user']) && isset($parts['pass'])) {
$href .= $parts['user'] . ':' . $parts['pass'] . '@';
}
$href .= $parts['host'];
if (isset($parts['port'])) {
$href .= ':' . $parts['port'];
}
$href .= dirname($parts['path'], 1).$path;
}
}
crawl_page($href, $depth - 1);
}
echo "URL:",$url,PHP_EOL,"CONTENT:",PHP_EOL,$dom->saveHTML(),PHP_EOL,PHP_EOL;
}
crawl_page("http://hobodave.com", 2);
Edit: I fixed some bugs from Tatu's version (works with relative URLs now).
Edit: I added a new bit of functionality that prevents it from following the same URL twice.
Edit: echoing output to STDOUT now so you can redirect it to whatever file you want
Edit: Fixed a bug pointed out by George in his answer. Relative urls will no longer append to the end of the url path, but overwrite it. Thanks to George for this. Note that George's answer doesn't account for any of: https, user, pass, or port. If you have the http PECL extension loaded this is quite simply done using http_build_url. Otherwise, I have to manually glue together using parse_url. Thanks again George.
In it's simplest form:
function crawl_page($url, $depth = 5) {
if($depth > 0) {
$html = file_get_contents($url);
preg_match_all('~<a.*?href="(.*?)".*?>~', $html, $matches);
foreach($matches[1] as $newurl) {
crawl_page($newurl, $depth - 1);
}
file_put_contents('results.txt', $newurl."\n\n".$html."\n\n", FILE_APPEND);
}
}
crawl_page('http://www.domain.com/index.php', 5);
That function will get contents from a page, then crawl all found links and save the contents to 'results.txt'. The functions accepts an second parameter, depth, which defines how long the links should be followed. Pass 1 there if you want to parse only links from the given page.
Why use PHP for this, when you can use wget, e.g.
wget -r -l 1 http://www.example.com
For how to parse the contents, see Best Methods to parse HTML and use the search function for examples. How to parse HTML has been answered multiple times before.
As mentioned, there are crawler frameworks all ready for customizing out there, but if what you're doing is as simple as you mentioned, you could make it from scratch pretty easily.
Scraping the links: http://www.phpro.org/examples/Get-Links-With-DOM.html
Dumping results to a file: http://www.tizag.com/phpT/filewrite.php
I came up with the following spider code. I adapted it a bit from the following: PHP - Is the there a safe way to perform deep recursion? it seems fairly rapid....
<?php
function spider( $base_url , $search_urls=array() ) {
$queue[] = $base_url;
$done = array();
$found_urls = array();
while($queue) {
$link = array_shift($queue);
if(!is_array($link)) {
$done[] = $link;
foreach( $search_urls as $s) { if (strstr( $link , $s )) { $found_urls[] = $link; } }
if( empty($search_urls)) { $found_urls[] = $link; }
if(!empty($link )) {
echo 'LINK:::'.$link;
$content = file_get_contents( $link );
//echo 'P:::'.$content;
preg_match_all('~<a.*?href="(.*?)".*?>~', $content, $sublink);
if (!in_array($sublink , $done) && !in_array($sublink , $queue) ) {
$queue[] = $sublink;
}
}
} else {
$result=array();
$return = array();
// flatten multi dimensional array of URLs to one dimensional.
while(count($link)) {
$value = array_shift($link);
if(is_array($value))
foreach($value as $sub)
$link[] = $sub;
else
$return[] = $value;
}
// now loop over one dimensional array.
foreach($return as $link) {
// echo 'L::'.$link;
// url may be in form <a href.. so extract what's in the href bit.
preg_match_all('/<a[^>]+href=([\'"])(?<href>.+?)\1[^>]*>/i', $link, $result);
if ( isset( $result['href'][0] )) { $link = $result['href'][0]; }
// add the new URL to the queue.
if( (!strstr( $link , "http")) && (!in_array($base_url.$link , $done)) && (!in_array($base_url.$link , $queue)) ) {
$queue[]=$base_url.$link;
} else {
if ( (strstr( $link , $base_url )) && (!in_array($base_url.$link , $done)) && (!in_array($base_url.$link , $queue)) ) {
$queue[] = $link;
}
}
}
}
}
return $found_urls;
}
$base_url = 'https://www.houseofcheese.co.uk/';
$search_urls = array( $base_url.'acatalog/' );
$done = spider( $base_url , $search_urls );
//
// RESULT
//
//
echo '<br /><br />';
echo 'RESULT:::';
foreach( $done as $r ) {
echo 'URL:::'.$r.'<br />';
}
Its worth remembering that when crawling external links (I do appreciate the OP relates to a users own page) you should be aware of robots.txt. I have found the following which will hopefully help http://www.the-art-of-web.com/php/parse-robots/.
I created a small class to grab data from the provided url, then extract html elements of your choice. The class makes use of CURL and DOMDocument.
php class:
class crawler {
public static $timeout = 2;
public static $agent = 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)';
public static function http_request($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, self::$agent);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, self::$timeout);
curl_setopt($ch, CURLOPT_TIMEOUT, self::$timeout);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
return $response;
}
public static function strip_whitespace($data) {
$data = preg_replace('/\s+/', ' ', $data);
return trim($data);
}
public static function extract_elements($tag, $data) {
$response = array();
$dom = new DOMDocument;
@$dom->loadHTML($data);
foreach ( $dom->getElementsByTagName($tag) as $index => $element ) {
$response[$index]['text'] = self::strip_whitespace($element->nodeValue);
foreach ( $element->attributes as $attribute ) {
$response[$index]['attributes'][strtolower($attribute->nodeName)] = self::strip_whitespace($attribute->nodeValue);
}
}
return $response;
}
}
example usage:
$data = crawler::http_request('https://stackoverflow.com/questions/2313107/how-do-i-make-a-simple-crawler-in-php');
$links = crawler::extract_elements('a', $data);
if ( count($links) > 0 ) {
file_put_contents('links.json', json_encode($links, JSON_PRETTY_PRINT));
}
example response:
[
{
"text": "Stack Overflow",
"attributes": {
"href": "https:\/\/stackoverflow.com",
"class": "-logo js-gps-track",
"data-gps-track": "top_nav.click({is_current:false, location:2, destination:8})"
}
},
{
"text": "Questions",
"attributes": {
"id": "nav-questions",
"href": "\/questions",
"class": "-link js-gps-track",
"data-gps-track": "top_nav.click({is_current:true, location:2, destination:1})"
}
},
{
"text": "Developer Jobs",
"attributes": {
"id": "nav-jobs",
"href": "\/jobs?med=site-ui&ref=jobs-tab",
"class": "-link js-gps-track",
"data-gps-track": "top_nav.click({is_current:false, location:2, destination:6})"
}
}
]
It's an old question. A lot of good things happened since then. Here are my two cents on this topic:
To accurately track the visited pages you have to normalize URI first. The normalization algorithm includes multiple steps:
- Sort query parameters. For example, the following URIs are equivalent after normalization:
GET http://www.example.com/query?id=111&cat=222 GET http://www.example.com/query?cat=222&id=111 Convert the empty path. Example:
http://example.org → http://example.org/Capitalize percent encoding. All letters within a percent-encoding triplet (e.g., "%3A") are case-insensitive. Example:
http://example.org/a%c2%B1b → http://example.org/a%C2%B1bRemove unnecessary dot-segments. Example:
http://example.org/../a/b/../c/./d.html → http://example.org/a/c/d.htmlPossibly some other normalization rules
- Sort query parameters. For example, the following URIs are equivalent after normalization:
Not only
<a>tag hashrefattribute,<area>tag has it too https://html.com/tags/area/. If you don't want to miss anything, you have to scrape<area>tag too.Track crawling progress. If the website is small, it is not a problem. Contrarily it might be very frustrating if you crawl half of the site and it failed. Consider using a database or a filesystem to store the progress.
Be kind to the site owners. If you are ever going to use your crawler outside of your website, you have to use delays. Without delays, the script is too fast and might significantly slow down some small sites. From sysadmins perspective, it looks like a DoS attack. A static delay between the requests will do the trick.
If you don't want to deal with that, try Crawlzone and let me know your feedback. Also, check out the article I wrote a while back https://www.codementor.io/zstate/this-is-how-i-crawl-n98s6myxm
Related