Run Scrapy from a Script - Issues with Item Import

user_18477534 0
Viewed 39

I am trying to run multiple spiders, to do this I want to run Scrapy as a script using what i saw in their documentation about CrawlerProcess. However, I want to use the Item I create with the different scrapy.Field objects which is inside the parent folder of where I am running the crawling script, here is a quick example to show what I am trying to achieve and how the folders are arranged with a project in scrapy:

enter image description here

The error i get is as follows:

enter image description here

After researching I cam across Relative Imports however doing this just results in this error message : enter image description here

I am Showing images So you can see how the folders are arranged to help understand my issue, any help would be appreciated.

scrapy web-scraping python
1 Answers

you are missing an __init__.py file in the root directory. It should be like this.

scrapy project structure

I assume you had an issue while creating the project.

For your case -

scrapy startproject example example to create the project

Then create a spider inside spiders directory.

cd to spiders directory and run the command (which you already did)

C:\users\.....\ExampleFolder\example\example\spiders>python examplespider.py

user_111053560
Related