pandas decoding a string returns NaN

user_2266957 0
Viewed 1138

I'm practicing on the kaggle news headline dataset: https://www.kaggle.com/aaron7sun/stocknews#Combined_News_DJIA.csv

df = pd.read_csv('./data/Combined_News_DJIA.csv')

When read the DataFrame of news headline, I get this off formatting of the series:

0       b"Georgia 'downs two Russian warplanes' as cou...
1       b'Why wont America & Nato help us? If they w...
2       b'Remember that adorable 9-year-old who sang a...
3       b' U.S. refuses Israel weapons to attack Iran:...
4       b'All the experts admit that we should legalis...

I tried using the following:

df['Series'].str.decode("utf-8")

However the output is a list of NaN. Any ideas? Would be great to implement on the whole DataFrame and not just one Series.

pandas decode python
1 Answers

You can't decode it from UTF-8 because it's already a string - not a byte-sequence.

The content of the file is indeed confusing: it contains strings that start with "b'..., which misleads the use to thinks it's bytes - but it's not.

If you run df.Top1[0], you'll see that it contains: 

'b"Georgia \'downs two Russian warplanes\' as countries move to brink of war"'

And type(df.Top1[0]) is just a string. Therefore - you can't decode it from UTF-8.

user_11055600
Related