When I use pd.concat, I encounter the specefic columns filled with NaN.
df_train_extended.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 7339 entries, 0 to 7338
Data columns (total 8 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 sentence_id 7339 non-null int64
1 person 7339 non-null object
2 sentence 7339 non-null object
3 scene 7339 non-null object
4 context 7339 non-null object
5 target_speaker_ctx 7339 non-null object
6 other_speaker_ctx 7339 non-null object
7 scene_sents 7339 non-null object
dtypes: int64(1), object(7)
memory usage: 458.8+ KB
df_train_extended_augmented.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 7339 entries, 0 to 7338
Data columns (total 8 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 sentence_id 7339 non-null int64
1 person 7339 non-null object
2 sentence 7339 non-null object
3 scene 7339 non-null object
4 context 7339 non-null object
5 target_speaker_ctx 7339 non-null object
6 other_speaker_ctx 7339 non-null object
7 scene_sents 7339 non-null object
dtypes: int64(1), object(7)
memory usage: 458.8+ KB
above tow dataframe have the name of sample column with Non-Null. But the result of the pd.concat is below.
df_data = pd.concat([df_train_extended, df_train_data_augmented], axis=0, ignore_index=True).reset_index(drop=True)
df_data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 14678 entries, 0 to 14677
Data columns (total 8 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 sentence_id 14678 non-null int64
1 person 14678 non-null object
2 sentence 14678 non-null object
3 scene 14678 non-null object
4 context 14678 non-null object
5 target_speaker_ctx 7339 non-null object
6 other_speaker_ctx 7339 non-null object
7 scene_sents 7339 non-null object
dtypes: int64(1), object(7)
memory usage: 917.5+ KB
I dont'n know why target_speaker_ctx, other_speaker_ctx, scene_sents of the df_data has NaN values.