I am trying to run a Scrapy spider inside my Django project's view. This is my project structure.
ScrapyPro .
├── ScrapyPro
│ ├── __init__.py
│ ├── items.py
│ ├── pipelines.py
│ ├── settings.py
│ └── spiders
│ ├── __init__.py
│ └── demo_spider.py
└── scrapy.cfg
│
WebCrawlerPro
├── WebCrawlerApp
│ ├── __init__.py
│ ├── urls.py
│ ├── admin.py
│ ├── apps.py
│ ├── models.py
│ ├── views.py
│
├── WebCrawlerPro
│ ├── __init__.py
│ ├── urls.py
│ ├── wsgi.py
│ ├── settings.py
└── scrapy.cfg
Here, ScrapyPro is scrapy project and WebCrawlerPro is Django project.
And here is the view from where I am trying to run a spider:
from django.shortcuts import render
import urllib
from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings
# Create your views here.
def home_crawl(request):
if request.method == "POST":
keyword = request.POST.get("search")
keyword = urllib.quote(keyword)
url =( "https://some_example.com/q/%s" % keyword)
crawler = CrawlerRunner(get_project_settings())
crawler.crawl('example', url=url)
crawler.start()
return render(request,'index.html',{"foo":"bar"})
But when I POST a request through the form in this view.
This error appears :
KeyError at /
'Spider not found: example'
And when I manually run the spider from ScrapyPro project by :
scrapy crawl example
It executes successfully. What am I doing wrong?