Passing kwargs into a scrapy contract

Viewed 66

I am trying to test a scrapy spider using scrapy contracts and want to know if there is a way to pass in kwargs. The spider takes a base_url kwarg to determine where to scrape. The base spider will fail verification if this kwarg is not provided.

The spider:

class ChildSpider(BaseSpider):

def __init__(self, **kwargs):
    super().__init__(**kwargs)
    # Use base url to set start urls

The base spider:

class BaseSpider(scrapy.Spider):

def __init__(self, base_url=False, **kwargs):
    if not base_url:
        raise ValueError("Base url must be provided")
    self.base_url = base_url.rstrip(" /")
    super().__init__(**kwargs)

The contract I added to it's parse() function:

"""This is a test.

    @url http://dummy-url.com
    @returns items <min limit>
    @returns requests <min limit>
    @scrapes <attributes>
    """

I have been trying to pass a kwarg in with this contract so the verification doesn't fail but have not had any luck. I know I can pass in kwargs using the -a flag for scrapy crawl. So far I have tried:

  • Appending -a base_url=<some url>" to scrapy check
  • Appending -a base_url=<some url>" to @url in the contract
  • Setting @cb_kwargs base_url=<some url> in the contract
0 Answers
Related