Is "utf-8-sig" suitable for decoding both UTF-8 and UTF-8 BOM?

user_12058289 0
Viewed 3318

I am using the Python CSV library to read two CSV files.

One is encoded with UTF-8-BOM, another is encoded with UTF-8. In my practice, I found that both files could be read by using "utf-8-sig" as encoding type:

from csv import reader 
with open(file_path, encoding='utf-8-sig') as csv_file:
    c_reader = reader(csv_file, delimiter=',')
    headers = next(c_reader)    
    for row in c_reader:
        print(row)

I want to confirm, is "utf-8-sig" suitable for decoding both UTF-8 and UTF-8 BOM? I am using Python version 3.6 and 3.7. Thanks for your answers!

utf-8 csv character-encoding python
1 Answers

The utf-8-sig codec will decode both utf-8-sig-encoded text and text encoded with the standard utf-8 encoding

>>> s = 'Straße'
>>> utf8_sig = s.encode('utf-8-sig')
>>> utf8 = s.encode('utf')
>>> print(utf8_sig.decode('utf-8-sig'))
Straße
>>> print(utf8.decode('utf-8-sig'))
Straße

From the codecs docs:

Before any of the Unicode characters is written to the file, a UTF-8 encoded BOM (which looks like this as a byte sequence: 0xef, 0xbb, 0xbf) is written ... On decoding utf-8-sig will skip those three bytes if they appear as the first three bytes in the file.

The utf-8-sig encoding in most common in Windows environments. If you're sharing files with users on mac or *nix systems, the standard utf-8 encoding is what they would expect to receive.

user_53209060
Related