When trying to translate the content of a column from NL to EN in a dataframe using the googletrans library, with the following code:
import pandas as pd
import numpy as np
from googletrans import Translator
#import newsarticles as dataframe
df_delpher = pd.read_csv (r'D:\file.csv')
#remove unnecessary column
df_delpher.drop(df_delpher.columns[0], axis=1, inplace=True)
#print number of rows
print(len(df_delpher.index))
#remove duplicate rows
df_delpher.drop_duplicates()
#re-print number of rows to check whether there where any duplicates.
print(len(df_delpher.index))
#In the case of the Delpher dataframe, no immediate duplicates where found, rows before and after are the same: 17284.
#remove first 6 charachters of the Date column so that only the year remains.
df_delpher['Date'] = df_delpher['Date'].str[6:]
#check the presence of every year in the dataframe to make sure all the years 1970-2018 are included
print(df_delpher.Date.unique())
#Years 1970 - 1995 are present in the Delpher dataframe
#count frequency of presence year in dataframe
print(df_delpher.Date.str.split(expand=True).stack().value_counts())
#count frequency of presence different papers in the datafrme
print(df_delpher.Newspaper.value_counts())
#remove all white spaces from content text
df_delpher.Content = df_delpher.Content.str.strip()
#remove rows with empty cells
#newspaper
df_delpher['Newspaper'].replace('', np.nan, inplace=True)
df_delpher.dropna(subset=['Newspaper'], inplace=True)
#Date
df_delpher['Date'].replace('', np.nan, inplace=True)
df_delpher.dropna(subset=['Date'], inplace=True)
#Content
df_delpher['Content'].replace('', np.nan, inplace=True)
df_delpher.dropna(subset=['Content'], inplace=True)
#remove punctuation from content text
df_delpher['Content'] = df_delpher['Content'].str.replace(r'[^\w\s]+', '')
#find out what object the columns are
print(df_delpher['Content'].dtypes)
#translate dataframe:
translator = Translator()
df_delpher['Content_en'] = df_delpher['Content'].apply(translator.translate, src='nl', dest='en')
I receive the following error:
TypeError: the JSON object must be str, bytes or bytearray, not NoneType
Selecting a value in the first row of the column "Content" returns type string:
type(df_delpher['Content'][0])
Out[39]: str
The dataframe is based upon a csv file and dtype of the column returns an object.
The column does contain values as shown in the linked image: df
Sample data:
<bound method NDFrame.head of Newspaper Date \
0 Trouw 1970
1 De Telegraaf 1970
2 Trouw 1970
3 De Volkskrant 1970
4 De Telegraaf 1970
... ... ...
15088 Algemeen Dagblad 2005
15089 Algemeen Dagblad 2005
15090 Algemeen Dagblad 2005
15091 Algemeen Dagblad 2005
15092 Algemeen Dagblad 2005
Content
0 HENGELO De Dikkersgroep verwacht voor 1969 ge...
1 HENGELO dinsdag De Dlkkarsgroep verwacht oor I...
2 DEN HAAG De tendens tot concentratie die over...
3 Van onze correspondent DEN HAAG 9 jan De bouw...
4 Van onze redactie wetenschappen AMSTERDAM vrij...
... ...
15088 ___ Wie zei tegen een roddelblad Ik ben dol op...
15089 ___ Wie zei tegen een roddelblad Ik ben dol op...
15090 ___ Wie zei tegen een roddelblad Ik ben dol op...
15091 ___ Wie zei tegen een roddelblad Ik ben dol op...
15092 ___ Wie zei tegen een roddelblad Ik ben dol op...
[14026 rows x 3 columns]>
How can I overcome this JSon object error?
Thanks.