How to iterate through XML children node in scrapy with python?

user_11141142 0
Viewed 310

I'm tying to scrape the comments on this page but I can't seem to get how to iterate throught the children of the node where the comments are wrapped and get the data points.

This is part of the hmtl: 

        <div class="comment">
            <div class="comment-user">
                <div class="comment-user-avatar">
                    <a href="https://www.picuki.com/profile/alexandera_300">
                        <img src="https://scontent-yyz1-1.cdninstagram.com/v/t51.2885-19/s150x150/98342975_2815537605343770_6875611169034338304_n.jpg?_nc_ht=scontent-yyz1-1.cdninstagram.com&amp;_nc_ohc=VjMtcOxXuaQAX_ZCqee&amp;oh=4cf78fecbadcb57a81672c6edecc15a2&amp;oe=5F02D580" alt="alexandera_300">
                    </a>
                </div>
                <div class="comment-user-nickname">
                    <a href="https://www.picuki.com/profile/alexandera_300">@alexandera_300</a>
                </div>
            </div>
            <div class="comment-text">
                #followforfollowback
            </div>
        </div>
        <div class="comment">
            <div class="comment-user">
                <div class="comment-user-avatar">
                    <a href="https://www.picuki.com/profile/coxlogan2008">
                        <img src="https://scontent-yyz1-1.cdninstagram.com/v/t51.2885-19/s150x150/101229634_275138197009045_1475918829270859776_n.jpg?_nc_ht=scontent-yyz1-1.cdninstagram.com&amp;_nc_ohc=e4gTZqQGpEAAX_7U-Q0&amp;oh=36b7f5d1a0d7069f2447f4a318edec7d&amp;oe=5F004A54" alt="coxlogan2008">
                    </a>
                </div>
                <div class="comment-user-nickname">
                    <a href="https://www.picuki.com/profile/coxlogan2008">@coxlogan2008</a>
                </div>
            </div>
            <div class="comment-text">
                
            </div>
        </div>

The python code snippet I'm using is this:

    def parse_post(self, response):
    img_url = response.meta['img_url']
    caption = response.meta['caption']

    url = response.meta['url']

    comments = response.xpath('//div[@id="commantsPlace"]/text()')
    for comment in comments:
        likes = response.xpath('.//span[@class="icon-thumbs-up-alt"]/text()').get()
        # need to put a regex here to get just the number value:
        num_of_comments = response.xpath('.//span[@id="commentsCount"]/text()').get()

        comment_user_name = comment.xpath('.//*[@class="comment-user-nickname"]/a/text()').get()
        comment_text = comment.xpath('.//*[@class="comment-text"]/text()').get()

        yield {'img_url': img_url,
               'caption': caption,
               'url': url,
               'likes': likes,
               'num_of_comments': num_of_comments,
               'comment_user_name': comment_user_name,
               'comment_text': comment_text}

However, when I run this, I only get the data for the first comment n times. Could someone help me with this? I don't get why the code doesn't iterate through the nodes.

Thanks in advance!

instagram scrapy screen-scraping python
1 Answers

I think your issue comes from your xpath for 'comments'. By taking only the text, you're not selecting the nodes. The following changes make it work for me:

# the likes & number of comments only have to be taken once, should not be part of the loop
likes = response.xpath('.//span[@class="icon-thumbs-up-alt"]/text()').get()
num_of_comments = response.xpath('.//span[@id="commentsCount"]/text()').get()
comments = response.xpath('//div[@id="commantsPlace"]/*[@class="comment"]')
for comment in comments:  
    comment_user_name = comment.xpath('.//*[@class="comment-user-nickname"]/a/text()').get()
    comment_text = comment.xpath('.//*[@class="comment-text"]/text()').get()
user_87940090
Related