Searching a product in a website - Scrapy

user_16587823 0
Viewed 146

I have a website, which has a search bar. I want the user to input a string and i want to search that specific string in the website and crawl the resulting page. One was I figured out to do so was take the search url and concatenate whatever the user enter so the url is "https://www.czone.com.pk/search.aspx?kw=" and if the user enters "ssd" i would concatenate it and that should be my start url. this is what i am trying to do here but it doesn't work so.


class ProductsSpider(scrapy.Spider):
    name="gaming"
    start_urls = ["https://www.czone.com.pk/search.aspx?kw="]
    product = input("Enter the item you are looking for")
    start_urls = start_urls[0] + product

    def parse(self,response):
       pass

However the spider gives an error upon execution [scrapy.core.engine] ERROR: Error while obtaining start requests ValueError: Missing scheme in request url: h

Is there anything i am doing wrong in the spider or is my entire approach to solve the problem is wrong?

scrapy web-scraping python
1 Answers

Try:

import scrapy

class ProductsSpider(scrapy.Spider):
    name="gaming"
    #url = "https://www.czone.com.pk/search.aspx?kw="
    product = input("Enter the item you are looking for")
    
    def start_requests(self):
        yield scrapy.Request(
            url = f'https://www.czone.com.pk/search.aspx?kw={product}',
            callback = self.parse
        )
    def parse(self,response):
       pass
user_128484110
Related