python scrapy execution sequences process request and the start urls

Viewed 25

I am wondering about the execution sequence of the python scrapy proxy middlewares, and the relationship with the length of start urls.

My case is:

1, I use proxy to visit the same domain,but several start urls

2, I was hoping the execution sequences just like this

the spider.py, get the list of start urls, then "process request" in the middlewares.py, then back the spider.py, execute the parse function.  When get to next start url, repeat this sequence.

3, but the reality is just like this:

spider.py to get the start urls(length:N) > process request N times in the middlewares.py > parse function N times.

my spider.py is as below:

class ProxytestspiderSpider(scrapy.Spider):
    name = 'proxytestspider'

    start_urls =[]
    for i in range(896, 898):
        start_url = 'https://xxxx.com/index.php?action=pages_view_main&active_action=repository_view_main_item_snippet&index_id={}&pn=1&count=20&order=7&lang=japanese&page_id=13&block_id=21'.format(i)
    start_urls.append(start_url)


    print(start_urls)   

    print('spider internal class')


    def parse(self, response):
        print('here in spider parse')   
        global total_result_list 
        result_list = []  
        blahblahblah......

in my middlewares.py the code is very simple as below:

class ChangeProxy(object):

def getIP(self): 
    self.random_proxy = 'http://' + '248.147.116.206:8180'
    print(
        'got a random_proxy %s' % self.random_proxy
    )

def process_request(self,request,spider):  
    self.getIP()
    request.meta['proxy'] = self.random_proxy

the settings :

DOWNLOADER_MIDDLEWARES = {

   # 'proxytest.middlewares.ProxytestDownloaderMiddleware': 543,
   'proxytest.middlewares.RotateUserAgentMiddleware': 541,
   'proxytest.middlewares.ChangeProxy': 542,
}

Or this is the way it works?! But it appears not.. Anyone help me thanks in advance!

0 Answers
Related