Python3 returning unexpected Unicode/ASCII

user_1709475 0
Viewed 47

Environment: Python 3.6, bs4

I am using the following code to 'webscrape' data from a local html file. The code is generally returning data as expected except instead of returning ü my code is returning ü.

I am saving the recovered data to a UTF-8 .csv file. The ü character is being displayed onto the console and into the .csv file. I am unsure where the problem lies. Is this a Unicode/charmap issue? My code? Or something else?

I would like the correct character to be returned, in this case ü

My code snippet...

#   https://stackoverflow.com/questions/50308840/error-extracting-text-from-website-attributeerror-nonetype-object-has-no-attr
    try:
        description_elemt = result.find("div", attrs = {'class':'product-desc'}).get_text(strip=True)
    except:
        traceback.print_exc()  #  debug info to console only
        description_elemt = 'No data given'
    #   replace 'Length' with 'length', this leaves 'Blade length: xxx mm' as one cell/element
    #   ref chapter 2.5, p45, Python Cookbook, 9781449340377
    description_elemt = description_elemt.replace('Length', 'length')
    description_elem = re.findall('[A-Z][^A-Z]*', description_elemt)
    #   gather the line items into one row ,separated by ,
    description_elem = ','.join(description_elem)```

Thanks guys.
web-scraping python-3.x python
1 Answers

I don't see anything in your example that displays the text on your console. You should also not be using ','.join(description_elem) to construct a CSV line. Note that the Unicode character ü is \xC3\xBC encoded as UTF-8. And in the ISO 8859-13 encoding \xC3 is à and \xBC is ¼. Other ISO 8859 encodings share one or both of those codepoint to glyph mappings. What this tells me is that the input you're parsing is not UTF-8, your program is not using a UTF-8 encoding, or you're looking at the output of your program using a non-UTF-8 locale. In other words, the problem is your code or the locale of your terminal emulator and/or shell.

user_42766470
Related