Scrapy - scraping comments skips the text after <br>

Viewed 1011

I am able to retrieve the text before the
tag but not the text after it.

This is the website that I am trying to scrape the comments from: http://hamusoku.com/archives/9589071.html#comments

Starting from some comments include the
tag which I think means that the user hit enter. Is there a way to get the text before and after the
tag as a single comment?

Here is a sample of the source code

<li="comment-body"> ==$0
    "
    愛の言葉も、この瞬間は辛い。"
    <br>
    "
    胸が締め付けられそうだ。"

This is my code:

import scrapy


class HamusoSpider(scrapy.Spider):
    name = 'hamuso'
    start_urls = ['http://hamusoku.com/archives/9589071.html#comments/']

    def parse(self, response):
        for com in response.css('li.comment-body'):
        item = {
        'comment': com.css('li::text').extract_first()
        }
        yield item

This is the output that I am getting in the shell:

{'comment': '\n\t\n\tかなしいなぁ'}
{'comment': '\n\t\n\t海老蔵…つらいな'}
{'comment': '\n\t\n\t海老蔵には頑張って欲しいな'}
{'comment': '\n\t\n\t御冥福をお祈りします'}
{'comment': '\n\t\n\t泣かすなや。'}
{'comment': '\n\t\n\t海老蔵これからしっかりせなアカンぞ'}
{'comment': '\n\t\n\t愛の言葉も、この瞬間は辛い。'}
{'comment': '\n\t\n\tただただ涙が止まらない会見だった'}

The last two comments both have a
tag and in both cases the second part of the comment is omitted.

I would really really appreciate any help with this.

1 Answers
Related