Convert HTML entities to Unicode and vice versa

You need to have BeautifulSoup.

from BeautifulSoup import BeautifulStoneSoup
import cgi

def HTMLEntitiesToUnicode(text):
    """Converts HTML entities to unicode.  For example '&' becomes '&'."""
    text = unicode(BeautifulStoneSoup(text, convertEntities=BeautifulStoneSoup.ALL_ENTITIES))
    return text

def unicodeToHTMLEntities(text):
    """Converts unicode to HTML entities.  For example '&' becomes '&'."""
    text = cgi.escape(text).encode('ascii', 'xmlcharrefreplace')
    return text

text = "&, ®, <, >, ¢, £, ¥, €, §, ©"

uni = HTMLEntitiesToUnicode(text)
htmlent = unicodeToHTMLEntities(uni)

print uni
print htmlent
# &, ®, <, >, ¢, £, ¥, €, §, ©
# &amp;, &#174;, &lt;, &gt;, &#162;, &#163;, &#165;, &#8364;, &#167;, &#169;

$ python3 -c "
> import html
> print(
>     html.unescape('&©—')
> )"
&©—

$ python3 -c "
> import html
> print(
>     html.escape('&©—')
> )"
&©—

$ python2 -c "
> from HTMLParser import HTMLParser
> print(
>     HTMLParser().unescape('&©—')
> )"
&©—

$ python2 -c "
> import cgi
> print(
>     cgi.escape('&©—')
> )"
&©—

HTML only strictly requires & (ampersand) and < (left angle bracket / less-than sign) to be escaped. https://html.spec.whatwg.org/multipage/parsing.html#data-state

For python3 use html.unescape():

import html
s = "&"
decoded = html.unescape(s)
# &

If someone like me is out there wondering why some entity numbers (codes) like ™ (for trademark symbol), € (for euro symbol) are not encoded properly, the reason is in ISO-8859-1 (aka Windows-1252) those characters are not defined.

Also note that, the default character set as of html5 is utf-8 it was ISO-8859-1 for html4

So, we will have to workaround somehow (find & replace those at first)

Reference (starting point) from Mozilla's documentation

https://developer.mozilla.org/en-US/docs/Web/Guide/Localizations_and_character_encodings

#!/usr/bin/env python3
import fileinput
import html

for line in fileinput.input():
    print(html.unescape(line.rstrip('\n')))