There are seemingly a million questions involving Python Unicode Errors where the ...ordinal [is] not in range(128). Seemingly, the vast majority involve Python 2.x.
I know about these errors because I am currently in encoding, decoding hell. For a side-project, I scrape web pages and attempt to normalize that text data, so that it doesn't appear on our site with crazy characters. To normalize the data, I rely on HTMLParser's HTMLParser() and entitydefs, as well as decoding the text from whatever its original form was (string.decode('[original encoding]', 'ignore')) and encoding it as UTF-8 (string.encode('utf-8', 'ignore')).
Yet, seemingly, there's always a site on which my best efforts fail, raising the same old UnicodeError: ASCII decoding error...ordinal not in range(128). It's so annoying.
I've read (here and here) that in Python 3 all text is Unicode. While I've read a lot about Unicode, because I'm not a software engineer, I don't know whether Unicode is objectively better (i.e., lower failure rate) than 2.x's default ascii encoding option. I have to think anything would be better, but I'd like if someone more expert and experienced could lend some perspective.
I'd like to know whether I should migrate to Python 3 for its (improved) processing of text scraped from the web. I am hoping that someone here can explain (or suggest resources that explain) the pros and cons of Python 3's approach to text processing. Is it better?? Is there someone who's dealt with my same problem who's already migrated to Python 3?? Would he/she recommend that I start using Python 3, if the 2to3 migration weren't an issue??
Thank you in advance for any assistance. I sure need it.