I'm learning Python and Django and for my personal project, I'm just playing with Scrapy for scraping website data and links.
It took me almost two days for setting up DjangoItem and writing spider but there is just one thing I can't solve by myself!
When I run the spider by "scrapy crawl WebspiderSpider", it runs and end without any errors. But the problem is the crawled data is not saved...I checked pipeline, items, settings, but it seems alright.
Please help me!
So this is my webspider.py
from zukkan_bot.items import WebItem
import datetime
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
class WebspiderSpider(CrawlSpider):
name = 'WebspiderSpider'
allowed_domains = ['example.com']
start_urls = ['http://example.com/']
xpath = {
'title': "//title/text()",
}
list_allow = [r'']
list_deny = [r'']
list_allow_parse = [r'']
list_deny_parse = [r'']
rules = (
Rule(LinkExtractor(
allow=list_allow,
deny=list_deny,
),
follow=True
),
Rule(LinkExtractor(
allow=list_allow_parse,
deny=list_deny_parse,
unique=True
),
callback='parse_items'
),
)
def parse_items(self, response):
item = WebItem()
item['title'] = response.xpath(self.xpath['title']).extract()[0]
item['link'] = response.url
item['date'] = datetime.datetime.utcnow() + datetime.timedelta(hours=9)
return WebItem(title=item['title'], link=item['link'], date=['date'])
And here is the items.py
from scrapy_djangoitem import DjangoItem
from zukkan.models import ExampleDotCom, Spider
class WebItem(DjangoItem):
django_model = Spider
This is the pipelines.py
class ZukkanBotPipeline(object):
def process_item(self, item, spider):
item.save()
return item
Finally, this is the log when I run the spider
scrapy crawl WebspiderSpider
/Users/XXXXXXX/PycharmProjects/begin/zukkan_bot/zukkan_bot/spiders/example.py:7: ScrapyDeprecationWarning: zukkan_bot.spiders.example.ExampleSpider inherits from deprecated class scrapy.spiders.BaseSpider, please inherit from scrapy.spiders.Spider. (warning only on first subclass, there may be others)
class ExampleSpider(BaseSpider):
2017-08-09 11:17:57 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: zukkan_bot)
2017-08-09 11:17:57 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'zukkan_bot', 'NEWSPIDER_MODULE': 'zukkan_bot.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['zukkan_bot.spiders']}
2017-08-09 11:17:57 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2017-08-09 11:17:57 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-08-09 11:17:57 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-08-09 11:17:57 [scrapy.middleware] INFO: Enabled item pipelines:
['zukkan_bot.pipelines.ZukkanBotPipeline']
2017-08-09 11:17:57 [scrapy.core.engine] INFO: Spider opened
2017-08-09 11:17:57 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-08-09 11:17:57 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-08-09 11:17:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://sample.com/robots.txt> (referer: None)
2017-08-09 11:17:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://sample.com/> (referer: None)
2017-08-09 11:17:57 [scrapy.core.engine] INFO: Closing spider (finished)
2017-08-09 11:17:57 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 436,
'downloader/request_count': 2,
'downloader/request_method_count/GET': 2,
'downloader/response_bytes': 5321,
'downloader/response_count': 2,
'downloader/response_status_count/200': 2,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 8, 9, 11, 17, 57, 988661),
'log_count/DEBUG': 3,
'log_count/INFO': 7,
'memusage/max': 55803904,
'memusage/startup': 55803904,
'response_received_count': 2,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2017, 8, 9, 11, 17, 57, 351302)}
2017-08-09 11:17:57 [scrapy.core.engine] INFO: Spider closed (finished)