Parse HTML page to get contents of <p> and <b> tags

user_7113854 0
Viewed 1593

There are lots of HTML pages which are structured as a sequence of such groups:

<p>
   <b> Keywords/Category:</b>
   "keyword_a, keyword_b"
</p>

The addresses of these pages are like https://some.page.org/year/0001, https://some.page.org/year/0002, etc.

How can I extract the keywords separately from each of such pages? I've tried to use BeautifulSoup, but unsuccessfully. I've only written the program that prints titles of groups (between <b> and </b>).

from bs4 import BeautifulSoup
from urllib2 import urlopen
import re
html_doc = urlopen('https://some.page.org/2018/1234').read()
soup = BeautifulSoup(html_doc)
for link in soup.find_all('a'):
    print 'https://some.page.org'+link.get('href')
for node in soup.findAll('b'):
    print ''.join(node.findAll(text=True))
web-crawler beautifulsoup html python
4 Answers

I can't test this without knowing the actual source code format but it seems you want the <p> tags text vaue:

for node in soup.findAll('p'):
    print(node.text)
    # or: keywords = node.text.split(', ')
    # print(keywords)
user_34343880

You need to split your string which in this case is url with /

And then you can choose chunks you want

For example if url is https://some.page.org/year/0001 i use split function to split url with / sign

it will convert it to array and then i choose what i need and again convert it to string with ''.join() method you can read about split method in this link

user_105705760

There are different ways to HTML parse the desired categories and keywords from this kind of HTML structure, but here is one of the "BeautifulSoup" ways to do it:

  • find b elements with a text which ends with a :
  • use .next_sibling to get to the next text node which contains keywords

Working example:

from bs4 import BeautifulSoup


data = """
<div>
    <p>
       <b> Category 1:</b>
       "keyword_a, keyword_b"
    </p>
    <p>
       <b> Category 2:</b>
       "keyword_c, keyword_d"
    </p>
</div>
"""

soup = BeautifulSoup(data, "html.parser")

for category in soup('b', text=lambda text: text and text.endswith(":")):
    keywords = category.next_sibling.strip('" \n').split(", ")

    print(category.get_text(strip=True), keywords)

Prints:

Category 1: ['keyword_a', 'keyword_b']
Category 2: ['keyword_c', 'keyword_d']
user_7718480

Assuming for each block

<p>
   <b> Keywords/Category:</b>
   "keyword_a, keyword_b"
</p>

you want to extract keyword_a and keyword_b for each Keywords/Category. So an example would be:

 <p>
    <b>Mammals</b>
    "elephant, rhino"
 </p>
 <p>
    <b>Birds</b>
    "hummingbird, ostrich"
 </p>

Once you have the HTML code, you can do:

from bs4 import BeautifulSoup

html = '''<p>
    <b>Mammals</b>
    "elephant, rhino"
    </p>
    <p>
    <b>Birds</b>
    "hummingbird, ostrich"
    </p>'''

soup = BeautifulSoup(html, 'html.parser')

p_elements = soup.find_all('p')
for p_element in p_elements:
    b_element = soup.find_all('b')[0]
    b_element.extract()
    category = b_element.text.strip()
    keywords = p_element.text.strip()
    keyword_a, keyword_b = keywords[1:-1].split(', ')
    print('Category:', category)
    print('Keyword A:', keyword_a)
    print('Keyword B:', keyword_b)

Which prints:

Category: Mammals
Keyword A: elephant
Keyword B: rhino
Category: Birds
Keyword A: hummingbird
Keyword B: ostrich
user_16210410
Related