How to clean HTML removing repeated paragraphs?

user_5691558 0
Viewed 98

I´m trying to clean an html file that has repeated paragraphs within body. Below I show the input file and expected output.

Input.html https://jsfiddle.net/97ptc0Lh/4/

Output.html https://jsfiddle.net/97ptc0Lh/1/

I've been trying with the following code using BeautifulSoup but I don´t know why is not working, since the resultant list CleanHtml contains the repeated elements (paragraphs) that I´d like to remove.

from bs4 import BeautifulSoup

fp = open("Input.html", "rb")
soup = BeautifulSoup(fp, "html5lib")

Uniques = set()
CleanHtml = []

for element in soup.html:
    if element not in Uniques:
        Uniques.add(element)
        CleanHtml.append(element)   

print (CleanHtml)

May someone help me to reach this goal please.

beautifulsoup parsing python
1 Answers

I think this should do it:

elms = []
for elem in soup.find_all('font'):
    if elem not in elms:
        elms.append(elem)
    else:
        target =elem.findParent().findParent()
        target.decompose()
print(soup.html)

This should get you your the desired output.

Edit:

To remove only for those paragraphs that have don't size 4 or 5, change the else block to

 else:
    if elem.attrs['size'] != "4" and elem.attrs['size'] !="5":
        target =elem.findParent().findParent()
        target.decompose()
user_94480900
Related