Continuously feed urls to a cluster of different scrapy spiders

My use-case is the following: I have a (redis) queue of urls that is filled externally. I want to crawl these urls with scrapy spiders. I have different spider classes for different domains, so depending on the url that is dequeued, I want to feed it to a different spider. There will be many urls in the queue, but for each url I will only crawl a single page. I won't follow any links. It's therefore important that there is little overhead in dispatching the urls to the spiders. The idea behind this setup is to be able to scale this horizontally, by deploying it to as many machines as I like, that all share the same redis queue.

I could create a crawler instance for each spider type and then use crawler.crawl with the next url. But the scrapy source code shows that this will always instantiate a new engine and spider, which seems like a lot of overhead to me.

crawler_instance = Crawler(MySpider, settings=get_project_settings())
crawler_instance.crawl()
crawler_process = CrawlerProcess(get_project_settings())
while next_url := get_next_url():
  # this seems to create new spider, engine, pipelines, ...
  crawler_process.crawl(crawler_instance, start_urls=[next_url])

My current solution is to have a main spider that internally keeps instances of the specific spiders and just passes the responses on to them.

class MainSpider(scrapy.Spider):
  def parse(self, response, spider_name):
    spider_instance = self._get_spider_instance(spider_name)
    return spider_instance.parse(response)

This works, but then the custom settings, pipelines, etc. of the specific spiders will be ignored and only the pipeline of the main spider runs. For example, it doesn't work if each spider type uses different DOWNLOAD_DELAYS or USER_AGENT.

I could also use something like scrapyd and just start a cluster of spiders, each spider with it's own url queue. But as far as I understand they would all run in separate processes and wouldn't respect settings like CONCURRENT_REQUESTS. I have around 50-100 different spider types, so running a separate process for each one doesn't seem like a good choice to me.

I'm wondering if what I'm trying even makes sense though: on the one hand, I want the spiders to share things like CONCURRENT_REQUESTS, but on the other hand each spider should respect it's individual DOWNLOAD_DELAY or ITEM_PIPELINES. Is that even possible? If so, how would I do it? Any help is appreciated, thanks.