High amount of Request to yield per parsed page blocks crawling

I have a crawl where every subpage contains 300+ links which I need to follow. The Crawl slows down after a minute or so / sometimes it crawls 0 pages / min.

If I rund the crawl with pages 10-50 links per page there same problem doesn't show.

I've configured 10 concurrent_requests and 10 processed items and reactor threadpool with 400. That means up to 3.000 yields per 10 processed items...

Logging shows that parse function takes 70+ seconds per page. Logging shows that the time is needed here comes from the yields (take up to 2 seconds per yield).

It seems scrapy waits until the engine? or something similar has done one task and is ready to process the new yield request? Adding the requests to scheduler don't take long so it seems to me that the yield is waiting for something other.

Any hints whats to tweak or whats going wrong?

Is it possible to yield the requests in bulk instead of yielding each on its own? Is it possible to add them to scheduler without yielding them?

Some additional Informations:

• No difference if I use scrapy-redis or just disk-base scheduler.
• Because of rendering javascript a download of a crawled page can take up to 10 seconds.
• Autothrottle is disabled
• if I give more cpu ressources it doesn't speed up things.

telnet-> est()

    time()-engine.start_time                        : 676.0599975585938
    engine.has_capacity()                           : False
    len(engine.downloader.active)                   : 7
    engine.scraper.is_idle()                        : False
    engine.spider.name                              : onetwothree
    engine.spider_is_idle(engine.spider)            : False
    engine.slot.closing                             : False
    len(engine.slot.inprogress)                     : 28
    len(engine.slot.scheduler.dqs or [])            : AttributeError (exception)
    len(engine.slot.scheduler.mqs)                  : AttributeError (exception)
    len(engine.scraper.slot.queue)                  : 0
    len(engine.scraper.slot.active)                 : 21
    engine.scraper.slot.active_size                 : 3878605
    engine.scraper.slot.itemproc_size               : 0
    engine.scraper.slot.needs_backout()             : False

AttributeError seems to come from scrapy-redis plugin, 
without scrapy counts up the pages to request in mgs.

est if len(engine.downloader.active) goes to 0

    time()-engine.start_time                        : 7236.464096784592
    engine.has_capacity()                           : False
    len(engine.downloader.active)                   : 0
    engine.scraper.is_idle()                        : False
    engine.spider.name                              : onetwothree
    engine.spider_is_idle(engine.spider)            : False
    engine.slot.closing                             : False
    len(engine.slot.inprogress)                     : 25
    len(engine.slot.scheduler.dqs or [])            : AttributeError (exception)
    len(engine.slot.scheduler.mqs)                  : AttributeError (exception)
    len(engine.scraper.slot.queue)                  : 0
    len(engine.scraper.slot.active)                 : 25
    engine.scraper.slot.active_size                 : 5357134
    engine.scraper.slot.itemproc_size               : 0
    engine.scraper.slot.needs_backout()             : True

Crawler Code:

    class robo2Spider(Spider):
        http_pass = None
        http_user = None
        dont_redirect = True
        start_urls = []
    
        def __init__(self, *args, **kwargs):
    
            # ... some config ...
    
            self.start_urls = self.custom_settings["TEST_URLS"]
            # Don't Follow links in test mode
    
        def start_requests(self):
            for url in self.start_urls:
                r = self.get_request(url)
                yield r
    
        def parse(self, response):
            # some extraction and co...
            yield from self.scrape_data(response)
    
        def scrape_data(self, response):
            start_time = time.time()
    
            # more extraction, build item
    
            extract_links = util.extract_links_from_response(response, self.query_pars_to_ignore)
            logging.info(
                "--- logging time 1: %s --- %s seconds ---" % (response.url, time.time() - start_time))
    
            request_links = []
            for link in extract_links:
    
                if (not link["nofollow"]) and (l.get_output_value("crawl_meta_nofollow") != "nofollow"):
                    r = self.get_request(link["url"])
                    request_links.append(r)
    
            yield from request_links
    
            logging.info(
                "--- logging time 2 (takes up to 70 sec): %s --- %s seconds ---" % (response.url, time.time() - start_time))
    
            yield l.load_item()
    
    
        def get_request(self, url, rit=None, splash_retry=None):
    
            # ... setting meta & co ...
            meta = {}
            splash_args = {}
            return SplashRequest(url=url, callback=self.parse, meta=meta,
                                 args=splash_args, http_status_from_error_code=True,
                                 endpoint='execute', slot_policy=scrapy_splash.SlotPolicy.SCRAPY_DEFAULT)

Same result if i do it that way:

    for link in extract_links:

        if (not link["nofollow"]) and (l.get_output_value("crawl_meta_nofollow") != "nofollow"):
            r = self.get_viu_request(link["url"])
            request_links.append(r)
            yield r