How to stop BeautifulSoup from decoding HTML entities into symbols

user_8109320 0
Viewed 297

I am trying to get all the links on a given website but is stuck with some problems about HTML entities. Here's my code that crawls websites using BeautifulSoup:

from bs4 import BeautifulSoup
import requests
    .
    .

baseRequest = requests.get("https://www.example.com", SOME_HEADER_SETTINGS)
soup = BeautifulSoup(baseRequest.content, "html.parser")
pageLinks = []

for anchor in soup.findAll("a"):
    pageLinks.append(anchor["href"])
    .
    .

print(pageLinks)

The code becomes problematic when it sees this kind of element:

<a href="./page?id=123&sect=2">Link</a>

Instead of printing ["./page?id=123&sect=2"], it treats the &sect part as an HTML entity and shows this in the console:

["./page?id=123§=2"]

Is there a solution to preventing the this?

beautifulsoup html python
1 Answers

Here is one

from bs4 import BeautifulSoup

soup = BeautifulSoup('<a href="./page?id=123&sect=2">Link</a>', "html.parser")
pageLinks = []

for anchor in soup.findAll("a"):
    pageLinks.append(anchor["href"])


uncoded = ''.join(i for i in pageLinks).encode('utf-8')
decoded = ''.join(map(lambda x: chr(ord(x)),''.join(i for i in pageLinks)))
print('uncoded =',uncoded)
print('decoded =',decoded)

output

uncoded = b'./page?id=123\xc2\xa7=2'
decoded = ./page?id=123§=2
user_135368930
Related