I am trying to test a scrapy spider using scrapy contracts and want to know if there is a way to pass in kwargs. The spider takes a base_url kwarg to determine where to scrape. The base spider will fail verification if this kwarg is not provided.
The spider:
class ChildSpider(BaseSpider):
def __init__(self, **kwargs):
super().__init__(**kwargs)
# Use base url to set start urls
The base spider:
class BaseSpider(scrapy.Spider):
def __init__(self, base_url=False, **kwargs):
if not base_url:
raise ValueError("Base url must be provided")
self.base_url = base_url.rstrip(" /")
super().__init__(**kwargs)
The contract I added to it's parse() function:
"""This is a test.
@url http://dummy-url.com
@returns items <min limit>
@returns requests <min limit>
@scrapes <attributes>
"""
I have been trying to pass a kwarg in with this contract so the verification doesn't fail but have not had any luck. I know I can pass in kwargs using the -a flag for scrapy crawl. So far I have tried:
- Appending
-a base_url=<some url>"toscrapy check - Appending
-a base_url=<some url>"to@urlin the contract - Setting
@cb_kwargs base_url=<some url>in the contract