How to write a check in python to see if file is valid UTF-8?

Viewed 17806

As stated in title, I would like to check in given file object (opened as binary stream) is valid UTF-8 file.

Anyone?

Thanks

3 Answers

If anyone needed a script to find all non utf-8 files in current dir: import os

def try_utf8(data):
    try:
        return data.decode('utf-8')
    except UnicodeDecodeError:
        return None


for root, _, files in os.walk('.'):
    if root.startswith('./.git'):
        continue
    for file in files:
        if file.endswith('.pyc'):
            continue
        path = os.path.join(root, file)
        with open(path, 'rb') as f:
            data = f.read()
            data = try_utf8(data)
            if data is None:
                print(path)
Related