I have a 800MB tar.gz in Farsi/Persian and I'm trying to open it into a dataframe and I use:
import pandas as pd
df = pd.read_csv('Arshasb_7k.tar.gz', compression='gzip', header=0, sep=' ', quotechar='"',
error_bad_lines=False)
df = df.drop(0)
df.rename(columns={0: "file_name", 1: "text"}, inplace=True)
df.head()
and I get this error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 4096: invalid start byte
I tried everything I found in here and it still doesn't work. Help me please.