How to extract all elements inside tables with BeautifulSoup?

user_5691558 0
Viewed 60

I have a big HTML file that was created merging several HTML files, so the structure is repeated N times <html>..</html> <html>..</html> like this:

<html>
<head>
<body>
    <TABLE>
    <TABLE>  
</body>
</html>

<html>
<head>
<body>
    <TABLE>
    <TABLE>  
</body>
</html>

<html>
<head>
<body>
    <TABLE>
    <TABLE>  
</body>
</html>

A more complete example is here https://jsfiddle.net/28du1omt/

I'd like to extract all kinds of elements within the second table of each html block in order to get this:

I say all elements, since second table could have inside paragraphs, tables, images, http links, etc (<p>, <img>, <tables>, etc)

<html>
  <head>
  </head>
  <body> 
        <p>&nbsp;</p>
        <p><font size="5" color="red">some text file 1</font></p>   
        <p><font size="4" color="purple">some text file 1</font></p>  
        <p>&nbsp;</p>
        <p>some text file 2</p>
        <p>&nbsp;</p>
        <p>some text file 3</p>                 
  </body>
</html>

How can I do this?

My current code looks all tables elements within file but I don´t know how to extract only elements inside each second table of each html block.

from bs4 import BeautifulSoup

fp = open("file.html", "rb")
soup = BeautifulSoup(fp, "html5lib")

tables = soup.find_all('table')

for tbl in tables:
    print(tbl.find_all())
beautifulsoup python-3.x html
3 Answers

My current code looks all tables elements within file but I don´t know how to extract only elements inside each second table of each html block.

You're on the right track already. What about iterating over html blocks first, and then only choosing the second tables appearing? Should be sufficient, no?

raw_html = """<html>
<head>
<body>
    <table></table>
    <table><test id="second_table">test text 1</test></table>
</body>
</html>
<html>
<head>
<body>
    <table></table>
    <table><test id="second_table">test text 2</test></table>
</body>
</html>
<html>
<head>
<body>
    <table></table>
    <table><test id="second_table">test text 3</test></table>
</body>
</html>"""

soup = BeautifulSoup(raw_html, "html.parser")

for html_block in soup.find_all("html"):
    interesting_table = html_block.find_all("table")[1]  # keep the second table only
    print(interesting_table)  # do what you want with these tables now

Output:

<table><test id="second_table">test text 1</test></table>
<table><test id="second_table">test text 2</test></table>
<table><test id="second_table">test text 3</test></table>
user_61596980

If you want html table into dataframe then you can try this 

import pandas as pd
import html5lib
url = 'file.html'
dfs = pd.read_html(url)
df = pd.concat(dfs)
df

you can go through read_html documentation as you can provide id's of table to get particular html table etc like attrs = {'id': 'table'}

user_75444780

Recommend a library that treats all HTML tags as strings.

from simplified_scrapy import SimplifiedDoc
html = '''
<html>
<head>
<body>
    <table></table>
    <table><test id="test_id">test text 1</test></table>
</body>
</html>
<html>
<head>
<body>
    <table></table>
    <table><test id="test_id">test text 2</test></table>
</body>
</html>
<html>
<head>
<body>
    <table></table>
    <table><test id="test_id">test text 3</test></table>
</body>
</html> 
'''
doc = SimplifiedDoc(html)
htmls = doc.htmls
for h in htmls:
  table = h.getElement('table',start='</table>') # Use start='</table>' to skip the first table
  print (table.test.text)
  print (table.select('test#test_id'))

Result:

test text 1
{'id': 'test_id', 'tag': 'test', 'html': 'test text 1'}
test text 2
{'id': 'test_id', 'tag': 'test', 'html': 'test text 2'}
test text 3
{'id': 'test_id', 'tag': 'test', 'html': 'test text 3'}
user_124558790
Related