I am trying to do two things:
- scrape a web page for a list of website
- Use the results of the first spider as input into a second spider which will scrape some other type of data.
I currently have step one completed, but am stuck trying to figure out how to use the output of the first spider as input into the second. Is this something that is even possible within the scrapy framework? Would it be easier to have these two spiders in separate projects/pipelines and maybe the second spider would just read from some file the first spider wrote its output to. Ty in advance for the assistance.