Cleaning text in a pandas column

Question

I have a problem in cleaning this dataset:

I would need to plot Text column in this dataset:

Date                 Account  Text
0   [28/07/2018]    412 ‎   [MAN]
1   [28/07/2018]    21403   ‎['photo', 'photo']
2   [28/07/2018]    3193    [Describe, Links']
3   [28/07/2018]    4239    ‎['photo']
4   [03/08/2018]    051     []
5   [03/08/2018]    21      []
6   [16/08/2018]    4124    ['XCUUE', 'HB']
7   [16/08/2018]    412     [program]
8   [16/08/2018]    410     []
9   [16/08/2018]    9543    [Links]

Specifically, I would need to work on the Text column. The rows within that column are the results of a tokenisation, so they are included in brackets. What I would like to do is to plot those words in a words cloud or a simply bar chart for studying their frequency. The problem that I am having is in the duplicate of the same word, for example 'photo' 'photo' when I try to plot the frequency.

My code is for preparing the dataset for a wordcloud.

comment_words = '' 
stopwords = remove_stop() # you can use a default one
  
for val in df['Text']: 
        
    tokens = val.split() 
      
    for i in range(len(tokens)): 
        tokens[i] = tokens[i].str.lower()
      
    comment_words += " ".join(tokens)+" "
  
wordcloud = WordCloud(width = 800, height = 800, 
                background_color ='white', 
                stopwords = stopwords, 
                min_font_size = 10).generate(comment_words)

I would really appreciate if you could have a look or try a different approach to fix this issues with duplicate words (frequency is important so I cannot drop any duplicate rows) and eventually with quotation mark '. Thank you

user_3195451 · Answer

There are many approaches you could take to solve your problem of removing duplicate words from within a given row. Here's one:

for val in df['Text']: 
        
    tokens = val.split()

    tokens = list(set(tokens))
      
    for i in range(len(tokens)): 
        tokens[i] = tokens[i].str.lower()
      
    comment_words += " ".join(tokens)+" "

user_423926 · Answer

If you want to remove duplicated with in records (there is no meaning to plot wordcloud by removing duplicates across records):

import pandas as pd
from wordcloud import WordCloud
from nltk.corpus import stopwords
import matplotlib.pyplot as plt

def plot(words):
  wordcloud = WordCloud(width = 800, height = 800, 
                  background_color ='white', 
                  stopwords = stopwords.words('english'), 
                  min_font_size = 10).generate(" ".join(words)) 

  plt.figure()
  plt.imshow(wordcloud, interpolation="bilinear")
  plt.axis("off")
  plt.show()

# Data
df = pd.DataFrame({'Text':
                   [
                    ['MAN'], 
                    ['photo', 'photo'], 
                    ['Describe', 'Links'],
                    ['photo'], [], [], 
                    ['XCUUE', 'HB'], 
                    ['program'], [], 
                    ['Links']
                   ]})

# Remove duplicates within records
words = df['Text'].apply(lambda x: list(set(x))).tolist()
words = [item for sublist in words for item in sublist]

plot(words)

To make word case-insensitive

plot([w.lower() for w in words])

Finally, if your dataframe records are untokenized text like below

df = pd.DataFrame({'Text': ['MAN', 'photo photo', 'Describe Links', 'photo', ' ', ' ', 'XCUUE HB', 'program', 'Links']})

Then you can

words = df['Text'].apply(lambda x: list(set(x.split()))).tolist()
words = [item for sublist in words for item in sublist]
plot([w.lower() for w in words])

Cleaning text in a pandas column

2 Answers