I wanted to spider a website and, if some text or a matching pattern is found in the HTML, get the URL(s) of the page(s).
Wrote the command
wget --recursive --spider site.com 2>&1 | sort | uniq | grep -oe 'www[^ ]*'
to get all the URLs thus far, but stuck as to how to output only those URLs that have a specified text. Any clues?
spider a website and, if some text or a matching pattern is found in the HTML
This is impossible with wget --spider. wget manual says that when you use --spider
When invoked with this option, Wget will behave as a Web spider, which means that it will not download the pages, just check that they are there. For example, you can use Wget to check your bookmarks:
wget --spider --force-html -i bookmarks.htmlThis feature needs much more work for Wget to get close to the functionality of real web spiders.
wget with --spider option does fetch response headers, which you can print following way
wget --spider --server-response http://www.example.com
which will contain information about file, for example Content-Length informing about file size, but not file content itself.