python - html - how to modify code by converting text outside of a tag into a tag

user_6728958 0
Viewed 544

How to replace/convert/correct a string representing tag into a tag?

I have below example where I need to clean some parts of the code and need to convert strings like </div> into the proper tags

html = """
    <html>
     <body>
      <div>
      &lt;/div&gt;  <----- how to convert the line into </div>
      <div class="first_class">
       <h1 id="Header_1">
        Header_1
       </h1>
      </div>
     </body>
    </html>  
"""

I tried

soup = BeautifulSoup(html,"lxml")

tag = soup.find(text="&lt;")
tag.replace_with("<")

print(soup.prettify())

but this logic doesn't work, the find function doesn't pick up the string. The fact that the text is outside of any tag makes it more difficult. How can this be achieved?

beautifulsoup html python
3 Answers

Using str.replace

In [3]: print(html.replace('&lt;', '<').replace('&gt;', '>'))

    <html>
     <body>
      <div>
      </div>
      <div class="first_class">
       <h1 id="Header_1">
        Header_1
       </h1>
      </div>
     </body>
    </html>

To place into BeautifulSoup from a file. Open the file first, replace the malformed text and then load the contents to BeautifulSoup. Something like this

with open('malformed.html') as f:
    malformed = f.read()

html = malformed.replace('&lt;', '<').replace('&gt;', '>')

soup = bs4.BeautifulSoup(html)
user_54913750

I think you need a function to decode them, such as unescape on html.parser.

from html.parser import HTMLParser

unescape = HTMLParser().unescape 
html = """
    <html>
     <body>
      <div>
      &lt;/div&gt;  <----- how to convert the line into </div>
      <div class="first_class">
       <h1 id="Header_1">
        Header_1
       </h1>
      </div>
     </body>
    </html>  
"""

print(unescape(html))

Output

<html>
 <body>
  <div>
  </div>  <----- how to convert the line into </div>
  <div class="first_class">
   <h1 id="Header_1">
    Header_1
   </h1>
  </div>
 </body>
</html>
user_93806940

Try using regular expressions instead.

Something like:

html = re.sub("&lt;", "<", html)

for less-than and 

html = re.sub("&gt;", ">", html)

for greater-than.

Make sure you import re first.

Edit: for reference on how to use re.sub - https://lzone.de/examples/Python%20re.sub

Edit2: After some further research it seems like str.replace() is faster, so you may want to use that instead.

user_51884300
Related