Scraping multiple urls and storing the corresponding data in separate files in scrapy

user_18059772 0
Viewed 430

I'm a university student working on a project and I'm pretty much new to using Scrapy. I've done as much research as I can stackoverflow/ yt but I cannot seem to integrate the ideas I have seen to what I'm trying to do. Basically, I have a list of urls that I need to scrape data from. I want to save/export the data scraped from each url to a corresponding json/csv file. The ultimate goal would then be to transfer those files to a database. I have managed to code the spider to get the data, however, I have to manually change the url in the spider class and export a file at a time. I cannot seem to figure out a way to automate it. This is my first time posting to stackoverflow, if you can help me this would be very much appreciated.

I have looked at pipelines/using open function with a write but I don't think I understood how to use them to export multiple files based on different urls.

I may not have formulated my question in the right way. I need to go through the urls list, scrape reviews and corresponding rating from the website and store those, ideally in a json/ database where i will access them later, clean the data and then feed them to a sentiment analysis model.

so for example,

  1. I have a list of urls that I need to go through (I have them stored in a csv file), check if they are "good" as some of the links don't work/ have no reviews

  2. scrape reviews and corresponding rating from the website and store those. json/ csv file for now as my plan is to add them to a database later(sql). [i'm working on figuring out if i can do it now itself though] as I will need to run a sentiment analysis model on those to predict the sentiment associated with a review and test the prediction against a given rating

note: because there is quite a lot of data to scrape, I was planning to clean it afterwards, thoughts?

I have now included a copy of the code. I started my manually changing the url myself and using the command line to export the file like: scrapy crawl spidey -O name_of_file.json

however this is not a very efficient way to go about it.

here is a snap of my code, https://imgur.com/GlgmB0q. I have added the item loader and multiple urls, before it was simply yield the items and a single url that i manually changed

Feel free to msg me if you can help, I would really much appreciate it. Twitter, discord etc...

scrapy python
1 Answers

You can use an item pipeline to split the output into multiple files. See below sample code.

import scrapy
from scrapy.exporters import CsvItemExporter
from urllib.parse import urlparse

class PerFilenameExportPipeline:
    def open_spider(self, spider):
        self.filename_to_exporter = {}

    def close_spider(self, spider):
        for exporter in self.filename_to_exporter.values():
            exporter.finish_exporting()

    def _exporter_for_item(self, item):
        filename = item.get('url').strip()
        if filename not in self.filename_to_exporter:
            f = open(f'{filename}.csv', 'wb')
            exporter = CsvItemExporter(f, export_empty_fields=True)
            exporter.start_exporting()
            self.filename_to_exporter[filename] = exporter
        return self.filename_to_exporter[filename]

    def process_item(self, item, spider):
        exporter = self._exporter_for_item(item)
        exporter.export_item(item)
        return item

class SampleSpider(scrapy.Spider):
    name = 'sample'
    start_urls = ['https://example.com', "https://www.scrapethissite.com/pages/simple", "https://stackoverflow.com"]

    custom_settings = {
        'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36',
        "ITEM_PIPELINES": {PerFilenameExportPipeline: 100}
    }

    def parse(self, response):
        parsed =  urlparse(response.url)
        yield {
            "url": parsed.netloc,
            "title": response.css("title::text").get()
        }
user_81799390
Related