Scrapy yield a Request, parse in the callback, but use the info in the original function

Viewed 6857

So I'm trying to test some webpages in scrapy, my idea is to yield a Request to the URLS that satisfy the condition, count the number of certain items on the page, and then within the original condition return True/False depending...

Here is some code to show what i mean:

def filter_categories:
    if condition:
        test = yield Request(url=link, callback = self.test_page, dont_filter=True)
        return (test, None)

def test_page(self, link):
    ... parse the response...
    return True/False depending

I have tried messing around with passing an item in the request, but no matter what the return line get's triggered before test_page is ever called...

So i guess my question becomes is there any way to pass data back to the filter_categories method in a synchronous way so that i can use the result of test_page to return whether or not my test is satisfied?

Any other ideas are also welcome.

3 Answers

you can use:

response.meta
response.body

yield from a function

to refactor your spider

Related