How to un-scrape first x number of p tag in a div and print rest of them?

Admin 1
Viewed 56

I'm have been spending past few days in learning of python. Today I come with a topic called web scraping. I'm trying to scrape all p tag inside a div except first 3 p tag. Since there is no class or id for the p tag I can't find way to un-scrape them

My code:

from bs4 import BeautifulSoup

data = '''<div class="one">
    <p style="color:red">Dummy Text</p>
    <p style="color:red">Unwanted Text</p>
    <p style="color:red">No Text</p>
    <p style="color:red">Lorem ipsum dolor sit amet</p>
    <p style="color:red">sed do eiusmod tempor incididunt</p>
    <p style="color:red">consectetur adipiscing elit</p>
    <p style="color:red">ut labore et dolore magna</p>
</div>'''

text = BeautifulSoup(data, 'html.parser')
for result in text.find_all('p'):
    print(result.get_text())

My output:

  • Dummy Text
  • Unwanted Text
  • No Text
  • Lorem ipsum dolor sit amet
  • sed do eiusmod tempor incididunt
  • consectetur adipiscing elit
  • ut labore et dolore magna

What I required:

  • Lorem ipsum dolor sit amet
  • sed do eiusmod tempor incididunt
  • consectetur adipiscing elit
  • ut labore et dolore magna

Since I'm new to SOF please mention in comment if I violated any of the guidelines.

web-scraping python
2 Answers

You should add the div strings to an array to store them. You should then remove the first three elements of the array. This can be done like this:

text = BeautifulSoup(data, 'html.parser')
p_list = []
for result in text.find_all('p'):
    p_list.append(result.get_text())
# this selects all elements from the list after the third
p_list = p_list[3:]

This will give you a list containing all of your p elements except the first three.

user_111366680

Method 1:

Find all p tag and index based on your requirement

soup = BeautifulSoup(data, 'html.parser')
soup.find_all("p")[3:]

Method 2:

soup = BeautifulSoup(data, 'html.parser')
data=soup.select_one("div > p:nth-of-type(2)")
data.find_next().find_next_siblings()

Breakdown of code:

first use select_one method and find p tag using css selector in which data returns <p style="color:red">No Text</p>

In last line we will find all next element with its siblings

Output (Same for both Metthod):

[<p style="color:red">Lorem ipsum dolor sit amet</p>,
 <p style="color:red">sed do eiusmod tempor incididunt</p>,
 <p style="color:red">consectetur adipiscing elit</p>,
 <p style="color:red">ut labore et dolore magna</p>]
user_115830700
Related