I was trying to use PunktWordTokenizer and it was occurred an error as below.
from nltk.tokenize.punkt import PunktWordTokenizer
And this gave the following error message.
Traceback (most recent call last): File "file", line 5, in <module>
from nltk.tokenize.punkt import PunktWordTokenizer ImportError: cannot import name PunktWordTokenizer
I've checked that nltk is installed and that PunkWordTokenzer is also installed using nltk.download(). Need some help for this.
PunktWordTokenizer was previously exposed to user but not any more. You can rather use WordPunctTokenizer.
from nltk.tokenize import WordPunctTokenizer
WordPunctTokenizer().tokenize(“text to tokenize”)
The difference is :
PunktWordTokenizer splits on punctuation, but keeps it with the word. Where as WordPunctTokenizer splits all punctuations into separate tokens.
For example, given Input: This’s a test
PunktWordTokenizer: [‘This’, “‘s”, ‘a’, ‘test’]
WordPunctTokenizer: [‘This’, “‘”, ‘s’, ‘a’, ‘test’]