Extracting raw HTML content (with tags) via beautifulsoup

user_16449781 0
Viewed 242

Using BeautifulSoup and Pandas, I am writing a module where I wish to extract full, raw HTML from pages/files and export the results to a spreadsheet. Here's an example:

Content.html file

<table>
<tbody>
<tr>
<td>Item 1</td>
</tr>
<tr data-name="item">
<td data-name="heading">Item 1</td>
<td data-name="content">Tagless Text in a cell.</td>
</tr>
<tr data-name="item">
<td data-name="heading">Item 2</td>
<td data-name="content">
  <p>Item with child elements.</p>
  <div>Second element.</div>
  <p>Third Element</p>
</td>
</tr>
<tr data-name="item">
<td data-name="heading">Item 3</td>
<td data-name="content">
<p>Item with direct and indirect child elements.
<ul>
  <li>Nested element 1</li>
  <li>Nested element 2</li>
  </ul>
</td>
</tr>
</tbody>
</table>

Python Script

from bs4 import BeautifulSoup
import csv
import pandas as pd
import numpy as np
import requests
import lxml

with open("content.html", "r") as source:
    #req = requests.get(url, headers)
    soup = BeautifulSoup(source, 'lxml')
    output = soup.findAll("td", attrs={"data-name": "content"})
    data = []
    for item in output:
        data.append(''.join(str(item)))

    df = pd.DataFrame(data, columns=["Content"])

    df.to_csv("data.csv", index=False)


# Output HTML code in file
#with open("code.html", "w") as f:
#    f.write(stuff)


print("Project Finished!")

This script currently works, but my output will contain the parent td element along with all of its content.

data.csv

Content
"<td data-name=""content"">Tagless Text in a cell.</td>"
"<td data-name=""content"">
<p>Item with child elements.</p>
<div>Second element.</div>
<p>Third Element</p>
</td>"
"<td data-name=""content"">
<p>Item with direct and indirect child elements.
</p><ul>
<li>Nested element 1</li>
<li>Nested element 2</li>
</ul>
</td>"

My ideal output would look like the following:

Content
"Tagless Text in a cell."
"<p>Item with child elements.</p>
<div>Second element.</div>
<p>Third Element</p>"
"
<p>Item with direct and indirect child elements.
</p><ul>
<li>Nested element 1</li>
<li>Nested element 2</li>
</ul>"

How can I achieve this? The closest I've been able to get so far either strips out all of the tags or keeps the tags, but outputs every child element in a list (which throws a "ValueError: X columns passed, passed data had Y columns" for elements with multiple items in said list).

pandas web-scraping beautifulsoup html python
1 Answers

You could try iterating over the .contents for each <td> tag, for example:

from bs4 import BeautifulSoup
import pandas as pd
import lxml

with open("content.html", "r") as source:
    #req = requests.get(url, headers)
    soup = BeautifulSoup(source, 'lxml')
    
    data = []
    for td in soup.find_all("td", attrs={"data-name": "content"}):
        data.append(''.join(str(el) for el in td.contents))

    df = pd.DataFrame(data, columns=["Content"])
    df.to_csv("data.csv", index=False)

print("Project Finished!")
user_49857330
Related