I am given a set of .txt files and I am trying to use the Keras Tokenizer to remove all punctuation and numbers. How can I use this module to do this? I have read the documentation here: https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer, but I am still unclear.
1 Answers
The Tokenizer class allows you to vectorize a text corpus by converting each text to a sequence of integers (each integer being the index of a token in a dictionary) like below.
from tensorflow.keras.preprocessing.text import Tokenizer
with open('data.txt') as f:
lines = f.readlines()
tokenizer = Tokenizer()
tokenizer.fit_on_texts(lines)
text=tokenizer.word_index
#prints the dictionary of words and their indices
print(text)
#print the words
print(list(text))
Output:
{'the': 1, 'words': 2, 'will': 3, 'be': 4, 'a': 5, 'is': 6, 'character': 7, 'maximum': 8, 'number': 9, 'of': 10, 'to': 11, 'keep': 12, 'based': 13, 'on': 14, 'word': 15, 'frequency': 16, 'only': 17, 'most': 18, 'common': 19, 'num': 20, '1': 21, 'kept': 22, 'string': 23, 'where': 24, 'each': 25, 'element': 26, 'that': 27, 'filtered': 28, 'from': 29, 'texts': 30, 'default': 31, 'all': 32, 'punctuation': 33, 'plus': 34, 'tabs': 35, 'and': 36, 'line': 37, 'breaks': 38, 'minus': 39, "'": 40}
['the', 'words', 'will', 'be', 'a', 'is', 'character', 'maximum', 'number', 'of', 'to', 'keep', 'based', 'on', 'word', 'frequency', 'only', 'most', 'common', 'num', '1', 'kept', 'string', 'where', 'each', 'element', 'that', 'filtered', 'from', 'texts', 'default', 'all', 'punctuation', 'plus', 'tabs', 'and', 'line', 'breaks', 'minus', "'"]
You can also use the RegexpTokenizer from nltk like below.
from nltk.tokenize import RegexpTokenizer
txt=[]
tokenizer = RegexpTokenizer(r'[A-Za-z]\w+')
for l in lines:
txt.append(tokenizer.tokenize(l))
print(txt)
Output:
[['the', 'maximum', 'number', 'of', 'words', 'to', 'keep', 'based', 'on', 'word', 'frequency', 'Only', 'the', 'most', 'common', 'num_words', 'words', 'will', 'be', 'kept'], ['string', 'where', 'each', 'element', 'is', 'character', 'that', 'will', 'be', 'filtered', 'from', 'the', 'texts', 'The', 'default', 'is', 'all', 'punctuation', 'plus', 'tabs', 'and', 'line', 'breaks', 'minus', 'the', 'character']]
Related