word count frequency except punctuation

user_12099096 0
Viewed 265

I want to count the 20 words that appear most frequently in the dataset, except for punctuation. So far the code I have tried is

dataset['token'].value_counts()[:20]

output :

,          88144
.          49109
dan        37283
di         33701
yang       29353
-LRB-      19843
-RRB-      19736
''         15906
``         15232
dengan     15231
pada       15021
dari       14900
tahun      13079
sebagai     9038
ini         8371
untuk       8297
dalam       8266
adalah      7950
menjadi     7414
oleh        5974

but there are still punctuation marks. how to do it so that the highest word frequency does not include the punctuation? Thanks..

python-3.x python
3 Answers

You have to do word preprocessing before counting then. Example :

import pandas as pd
import re

REPLACE_BY_SPACE_RE = re.compile('[/(){}\[\]\|@,;]')
BAD_SYMBOLS_RE = re.compile('[^0-9a-z #+_]')

def clean_text(text):
    text = text.lower()
    text = REPLACE_BY_SPACE_RE.sub(' ', text)
    text = BAD_SYMBOLS_RE.sub('', text)
    
    return text

ft = pd.read_json(path_of_file) # read ur file in pandas df
ft = ft.apply(clean_text)

Good luck

user_50384970

A pure Python solution.

  1. Remove everything but word characters from the text
  2. Give all the words to a Counter
  3. Ask for the 20 most common.
from collections import Counter

def get_most_common(text, n=20):
    processed_text = "".join(t for t in text.lower() if t.isalpha() or t == " ")
    counter = Counter(processed_text.split())
    return counter.most_common(n)
user_19041460

Try following code:

Assuming you are not interested in only punctuation values. If there is both punctuation and words, it would be counted.

dataset['token'][dataset['token'].str.contains(r'\w+')].value_counts()[:20]
user_92691630
Related