As stated in title, I would like to check in given file object (opened as binary stream) is valid UTF-8 file.
Anyone?
Thanks
As stated in title, I would like to check in given file object (opened as binary stream) is valid UTF-8 file.
Anyone?
Thanks
If anyone needed a script to find all non utf-8 files in current dir: import os
def try_utf8(data):
try:
return data.decode('utf-8')
except UnicodeDecodeError:
return None
for root, _, files in os.walk('.'):
if root.startswith('./.git'):
continue
for file in files:
if file.endswith('.pyc'):
continue
path = os.path.join(root, file)
with open(path, 'rb') as f:
data = f.read()
data = try_utf8(data)
if data is None:
print(path)