How to crawl from base url to another url in scrapy using request library

user_8978191 0
Viewed 989
import scrapy
from scrapy.selector import Selector
from scrapy.contrib.spiders import CrawlSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.http import Request


class NextlinkSpider(scrapy.Spider):
    name = 'nextlink'
    allowed_domains = ['www.shanazrafiq.com/']
    start_urls = ['https://www.shanazrafiq.com/']

    def parse(self, response):
        yield Request(url='https://www.shanazrafiq.com/p/recipes.html',callback = self.parse_dir_contents)

Error :AttributeError: 'NextlinkSpider' object has no attribute 'parse_dir_contents'

trying to crawl to another url from base url

I wanted to get the recipes page tag content by navigating from first page but unable to call the callback function which say me attribute not found error while doing so. Help me in fixing this . Thanks in advance 

def parse_dir_contents(self, response): # second function to be called.
    sel = Selector(response)
    title_name=sel.css("div.widget HTML h2::text").extract()
    print title_name
scrapy python
2 Answers

Guessing by the error message you have provided, your NextlinkSpider class doesn't have the parse_dir_contents method. I am guessing you have defined a function with the same name outside of the class (or somewhere else).

I'd suggest trying "glueing" these two together:

class NextlinkSpider(scrapy.Spider):
    name = 'nextlink'
    allowed_domains = ['www.shanazrafiq.com/']
    start_urls = ['https://www.shanazrafiq.com/']

    def parse(self, response):
        yield Request(url='https://www.shanazrafiq.com/p/recipes.html',callback = self.parse_dir_contents)

    def parse_dir_contents(self, response): # second function to be called.
       sel = Selector(response)
       title_name=sel.css("div.widget HTML h2::text").extract()
       print title_name

No changes in the code (other than indentation)

user_1185870

I've made a few changes and the following code should get you on the right track. This will use the scrapy.CrawlSpider and follow all recipe links on the start_urls page. It will extract the title, url, and image url on each individual recipe page. Hope this helps:

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from scrapy.http import Request

class NextlinkItem(scrapy.Item):
    title_name = scrapy.Field()
    url = scrapy.Field()
    image_urls = scrapy.Field()


class NextlinkSpider(CrawlSpider):
    name = 'nextlink'
    allowed_domains = ['shanazrafiq.com']
    start_urls = ['https://www.shanazrafiq.com/p/recipes.html']

    # Xpath for selecting links to follow
    xp = '//div[contains(@class, "post-body")]/div/div/div/h4/ul/li/a'

    rules = (
        Rule(LinkExtractor(restrict_xpaths=xp), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        item = NextlinkItem()
        item['title_name'] = response.xpath('//div[contains(@class, "post-outer")]/div/h3/text()').extract_first().strip()
        item['url'] = response.url
        item['image_urls'] = response.xpath('//div[contains(@class, "post-outer")]/div/meta[@itemprop="image_url"]/@content').extract()
        yield item
user_89642970
Related