Cannot import my own module in scrapy crawler

user_8551363 0
Viewed 1484

I'm writing a crawler using Scrapy. I've built a crawler and it works very well.

Now I want to create my own modules, but I always receive this error:

File "D:\Projects\bitbucket\terranoha\crawl1\crawl1\spiders\samplecrawler.py", line 4, in import moduletest

ModuleNotFoundError: No module named 'moduletest'

The code is: 

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
import moduletest

class SamplecrawlerSpider(CrawlSpider):
    # [...]

I am running: scrapy crawl --nolog samplecrawler. I'm on Windows 10.

My project structure is:

enter image description here

scrapy python
4 Answers

You can do several things:

First

from crawl1.spiders.moduletest import mythings

As suggested by @elRuLL

Second

from .moduletest import mythings

This generally a bad and brittle solution but possible.

Third

You can package it as package and do.

init.py:

from spiders.moduletest import *
__all__ = [<Put your classes, methods, etc here>]

samplecrawler.py

import moduletest
user_44733860

you need to include the full module path:

from crawl1.spiders.moduletest import mythings
user_8589130

you have to include the name of the folder as the module name

import crawl1.spiders.moduletest
user_33325810

Found, after some hours:

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
import crawl1.spiders.moduletest

class SamplecrawlerSpider(CrawlSpider):

import crawl1.spiders.moduletest

user_85513630
Related