I'll try my best to explain what's going on because I'm completely lost.
print("check")
process = CrawlerProcess(get_project_settings())
process.crawl('BoxOfficeSpider', df=movie_data)
process.start()
So I've narrowed down the problem to this specific block of code. What happens is this: If process = CrawlerProcess(), it executes as intended, going line by line down and then ending. However, when I pass in get_project_settings(), it for some reason runs the 2nd line, then RERUNS the first line! As in, it prints check twice. Using debug I've confirmed it is in fact moving the head (may not be the right term but im trying to say the line being executed) back exactly one executable line (meaning comments do not affect it).
Here is my settings file
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'extractData'
SPIDER_MODULES = ['extractData.spiders']
NEWSPIDER_MODULE = 'extractData.spiders'
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 4
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
AUTOTHROTTLE_ENABLED = True
# The initial download delay
AUTOTHROTTLE_START_DELAY = 1
# The maximum download delay to be set in case of high latencies
AUTOTHROTTLE_MAX_DELAY = 3
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# LOG_LEVEL = 'INFO'
LOG_ENABLED = True
I've done a lot of debugging for this issue, but I am confident it has exclusively to do with CrawlerProcess and get_project_settings(), as removing get_project_settings() completely fixes the issue (except I need global settings) but my dives into the documentation have not revealed why. Any help is appreciated.
If it provides more information, I have 2 spiders in my spiders folder, but only 1 is actually called. The other is correctly set up settings wise so it shouldn't be interfering, just thought I'd add that in case it's somehow affecting things. Here it is:
class FilmRatingsSpider(scrapy.Spider):
name = "FilmRatingsSpider"
allowed_domains = ["filmratings.com"]
start_urls = ['filmratings.com']
custom_settings = {
'LOG_FILE':'film_ratings_spider.log',
'ITEM_PIPELINES':{'extractData.pipelines.FilmRatingsPipeline': 400}
}
def parse(self, response, tconst):
pass
def __init__(self, df):
pass
Since the issue occurs before even reaches the 3rd line, I think it definitely has to do with CrawlerProcess or more likely settings.py, just don't know how or why.