I'm trying to use Scrapy to get all links on websites where the "DNS lookup failed".
The problem is, every website without any errors are print on the parse_obj method but when an url return DNS lookup failed, the callback parse_obj is not call.
I want to get all domain with the error "DNS lookup failed", how can I do that ?
Logs :
2016-03-08 12:55:12 [scrapy] INFO: Spider opened
2016-03-08 12:55:12 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-03-08 12:55:12 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-03-08 12:55:12 [scrapy] DEBUG: Crawled (200) <GET http://domain.com> (referer: None)
2016-03-08 12:55:12 [scrapy] DEBUG: Retrying <GET http://expired-domain.com/> (failed 1 times): DNS lookup failed: address 'expired-domain.com' not found: [Errno 11001] getaddrinfo failed.
Code :
class MyItem(Item):
url= Field()
class someSpider(CrawlSpider):
name = 'Crawler'
start_urls = ['http://domain.com']
rules = (Rule(LxmlLinkExtractor(allow=()), callback='parse_obj', follow=True),)
def parse_obj(self, response):
item = MyItem()
item['url'] = []
for link in LxmlLinkExtractor(allow=()).extract_links(response):
parsed_uri = urlparse(link.url)
url = '{uri.scheme}://{uri.netloc}/'.format(uri=parsed_uri)
print url