I am parsing a .html file using BeautifulSoup4 doing the following:
data = [item.text.strip() for item in soup.find_all('span')]
The code takes all the items in a given table and stores into data. I noticed some of the elements in the data contain texts what seems like html entity encoding. An example element:
data[5] stores 'CSCI-GA.1144-\u200b001'
the text I expected was just CSCI-GA.1144-001'
In the html file, I find it as 'CSCI-GA.1144-​001'
Why does it show differently when I parse, vs when I inspect the html code? And how do I parse the data so it does not take into account these encodings? Is there a way to exclude?