Spider not found: Django Scrapy integration

Viewed 144

I am trying to run a Scrapy spider inside my Django project's view. This is my project structure.

ScrapyPro       .
    ├── ScrapyPro
    │   ├── __init__.py
    │   ├── items.py
    │   ├── pipelines.py
    │   ├── settings.py
    │   └── spiders
    │       ├── __init__.py
    │       └── demo_spider.py
    └── scrapy.cfg
    │  
WebCrawlerPro   
    ├── WebCrawlerApp
    │   ├── __init__.py
    │   ├── urls.py
    │   ├── admin.py
    │   ├── apps.py
    │   ├── models.py
    │   ├── views.py
    │
    ├── WebCrawlerPro
    │   ├── __init__.py
    │   ├── urls.py
    │   ├── wsgi.py
    │   ├── settings.py
    └── scrapy.cfg

Here, ScrapyPro is scrapy project and WebCrawlerPro is Django project.

And here is the view from where I am trying to run a spider:

from django.shortcuts import render
import urllib
from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings

# Create your views here.
def home_crawl(request):

    if request.method == "POST":
        keyword = request.POST.get("search")
        keyword = urllib.quote(keyword)
        url =( "https://some_example.com/q/%s" % keyword)
        crawler = CrawlerRunner(get_project_settings())
        crawler.crawl('example', url=url)
        crawler.start()

    return render(request,'index.html',{"foo":"bar"})

But when I POST a request through the form in this view.

This error appears :

KeyError at /
'Spider not found: example'

And when I manually run the spider from ScrapyPro project by :

scrapy crawl example

It executes successfully. What am I doing wrong?

0 Answers
Related