Why does Scrapy CrawlerProcess give control to line before it, causing it and the previous line to run 2-3 times?

Viewed 71

I'll try my best to explain what's going on because I'm completely lost.

print("check")
process = CrawlerProcess(get_project_settings())
process.crawl('BoxOfficeSpider', df=movie_data)    
process.start()

So I've narrowed down the problem to this specific block of code. What happens is this: If process = CrawlerProcess(), it executes as intended, going line by line down and then ending. However, when I pass in get_project_settings(), it for some reason runs the 2nd line, then RERUNS the first line! As in, it prints check twice. Using debug I've confirmed it is in fact moving the head (may not be the right term but im trying to say the line being executed) back exactly one executable line (meaning comments do not affect it).

Here is my settings file

#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'extractData'

SPIDER_MODULES = ['extractData.spiders']
NEWSPIDER_MODULE = 'extractData.spiders'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 4

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
AUTOTHROTTLE_ENABLED = True
# The initial download delay
AUTOTHROTTLE_START_DELAY = 1
# The maximum download delay to be set in case of high latencies
AUTOTHROTTLE_MAX_DELAY = 3
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# LOG_LEVEL = 'INFO'
LOG_ENABLED = True

I've done a lot of debugging for this issue, but I am confident it has exclusively to do with CrawlerProcess and get_project_settings(), as removing get_project_settings() completely fixes the issue (except I need global settings) but my dives into the documentation have not revealed why. Any help is appreciated.

If it provides more information, I have 2 spiders in my spiders folder, but only 1 is actually called. The other is correctly set up settings wise so it shouldn't be interfering, just thought I'd add that in case it's somehow affecting things. Here it is:

class FilmRatingsSpider(scrapy.Spider):
    name = "FilmRatingsSpider"
    allowed_domains = ["filmratings.com"]
    start_urls = ['filmratings.com']
    custom_settings = {
        'LOG_FILE':'film_ratings_spider.log',
        'ITEM_PIPELINES':{'extractData.pipelines.FilmRatingsPipeline': 400}
    }
    def parse(self, response, tconst):
        pass

    def __init__(self, df):
        pass

Since the issue occurs before even reaches the 3rd line, I think it definitely has to do with CrawlerProcess or more likely settings.py, just don't know how or why.

1 Answers

So I figured it out, and I feel really dumb. For some reason that I haven't figured out yet, the entire python file was running twice when I ran get_project_settings(). It's probably because it imports my init.py file (the one I call get_project_settings from) and that caused it to run again. Adding a simple if __name__ == '__main__' and putting all the code into that fixes it completely.

Related