Catch UnicodeDecodeError exception while reading file line by line in Python 3

user_4378613 0
Viewed 5615

Consider the following code:

with open('file.txt', 'r') as f:
    for line in f:
        print(line)

In Python 3, the interpreter tries to decode the strings it reads, which might lead to exceptions like UnicodeDecodeError. These can of course be caught with a try ... except block around the whole loop, but I would like to handle them on a per-line basis.

Question: Is there a way to directly catch and handle exceptions for each line that is read? Hopefully without changing the simple syntax of iterating over the file too much?

io python-3.x exception python
4 Answers

The Pythonic way is probably to register an error handler with codecs.register_error_handler('special', handler) and declare it in the open function:

with open('file.txt', 'r', error='special') as f:
    ...

That way if there is an offending line, the handler will the called with the UnicodeDecodeError, and will be able to return a replacement string or re-raise the error.

If you want a more evident processing, an alternate way would be to open the file in binary mode and explicitely decode each line:

with open('file.txt', 'rb') as f:
    for bline in f:
        try:
            line = bline.decode()
            print(line)
        except UnicodeDecodeError as e:
            # process error
user_35452730

Instead of employing a for loop, you could call next on the file-iterator yourself and catch the StopIteration manually.

with open('file.txt', 'r') as f:
    while True:
        try:
            line = next(f)
            # code
        except StopIteration:
            break
        except UnicodeDecodeError:
            # code
user_36200030

Basing on @SergeBallesta's answer. Here's the simplest thing that should work.

Instead of open(), use codecs.open(..., errors='your choice'). It can handle Unicode errors for you.

The list of error handler names includes

'replace': "Replace with a suitable replacement marker; Python will use the official U+FFFD REPLACEMENT CHARACTER for the built-in codecs on decoding, and ‘?’ on encoding"

which should handle the error and add a marker "there was something invalid here" to the text.

import codecs

# ...

# instead of open('filename.txt'), do:
with codecs.open('filename.txt', 'rb', 'utf-8', errors='replace') as f:
    for line in f:
        # ....
user_2077910

Place your try-except catch inside the for loop, like so:

with open('file.txt', 'r') as f:
    for line in f:
      try:  
        print(line)
      except:
        print("uh oh")
        # continue
user_25839970
Related