How to prevent BeautifulSoup from adding extra doctype entry

user_9183027 0
Viewed 40

If I read an html file and load it with bs4, I get an extra doctype entry. How can I prevent it?

HTML Code

<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
<body>
<p>
text body
</p>
</body>
</html>

This is how the file is processed

from bs4 import BeautifulSoup

page = urllib.urlopen(file_name).read()
page_soup = BeautifulSoup(page, 'html.parser')

The resulting HTML

<!DOCTYPE doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
<body>
<p>
text body
</p>
</body>
</html>
beautifulsoup python
2 Answers

Perhaps the issue is not with BS as I am not able to reproduce the problem.

Running this

from bs4 import BeautifulSoup
import urllib.request


file_name = 'file:///C:/Users/tang/MathScripts/t.html'
page = urllib.request.urlopen(file_name).read()
soup = BeautifulSoup(page, 'html.parser')
print(soup)

I get

<!DOCTYPE html public "-//w3c//dtd html 4.0 transitional//en">

<html>
<body>
<p>
text body
</p>
</body>
</html>
user_136847890

It looks like doctype string is case insensitive in HTML spec, but case sensitive in XML spec.

It is explained very well in this post: "Uppercase or lowercase doctype?".

Based on this information, I think BeautifulSoup is not handling html doctype string properly.

I changed my code as below and it works fine now.

page = urllib.urlopen(file_name).read()
# case insensitive replace to consider all case permutations
page = re.sub('<!doctype', '<!DOCTYPE', page, flags=re.IGNORECASE)
page_soup = BeautifulSoup(page, 'html.parser')

I am not so sure if the html specification has been updated or not.

Please post comment if you have more information to share.

user_91830270
Related