word tokenization takes too much time to run

Question

I use Pythainlp package to tokenize my Thai language data for doing sentiment analysis. first, I build a function to add new words set and tokenize it

from pythainlp.corpus.common import thai_words
from pythainlp.util import dict_trie
from pythainlp import word_tokenize

def text_tokenize(Mention):
 new_words = {'คนละครึ่ง', 'ยืนยันตัวตน', 'เติมเงิน', 'เราชนะ', 'เป๋าตัง', 'แอปเป๋าตัง'}
 words = new_words.union(thai_words())
 custom_dictionary_trie = dict_trie(words)
 dataa = word_tokenize(Mention, custom_dict=custom_dictionary_trie, keep_whitespace=False)
 return dataa

after that I apply it within my text_process function which including remove punctuation and stop words.

puncuations = '''.?!,;:-_[]()'/<>{}\@#$&%~*ๆฯ'''
from pythainlp import word_tokenize
def text_process(Mention):
  final = "".join(u for u in Mention if u not in puncuations and ('ๆ', 'ฯ'))
  final = text_tokenize(final)
  final = " ".join(word for word in final)
  final = " ".join(word for word in final.split() if word.lower not in thai_stopwords)
  return final

dff['text_tokens'] = dff['Mention'].apply(text_process) 
dff

the point is it takes too long to run this function. it took 17 minutes and still not finished. I tried to replace final = text_tokenize(final) with final = word_tokenize(final)

and it took just 2 minutes but I can't no longer use it because I need to add new custom dictionary. I know there is something wrong but really don't know how to fix it

I am new to python and nlp so please help. Ps. sorry for my broken English

user_7714681 · Answer

I am not familiar with Thai language, but assume that for tokenization you can also use language agnostic tokenization tools.

If you want to perform word tokenization, try the example below:

from nltk.tokenize import word_tokenize
s = '''This is the text I want to tokenize'''
word_tokenize(s)

>>> ['This', 'is', 'the', 'text', 'I', 'want', 'to', 'tokenize']

word tokenization takes too much time to run

1 Answers