Is this simply an error from my code or is NLTK really bad at detecting words?

user_13545269 0
Viewed 55

I just started developing a very simple program that gets a txt file and tells you the misspelled words according to it. I looked up what would be the best program to use and I read that NLTK and use 'Words'. I did it and noticed that it is not doing its job correctly or maybe I'm not doing something correctly and it's actually my fault but can someone please check it out. 

from nltk.corpus import words
setwords = set(words.words())

def prompt():
    userinput = input("File to Evaluate: ").strip()
    with open(userinput, 'r') as file:
        words = file.read()
        return words

def main():
    error_list = []
    words = prompt()
    words_splitted = words.split()
    for i in words_splitted:
        if i in setwords:
            pass
        elif i not in setwords:
            error_list.append(i)

    print(f"We're not sure these words exist: {error_list}")





if __name__ == '__main__':
    main()

The program runs fine but please give me some help in figuring out if NLTK is actually bad at detecting words or it's failing in my program. I'm using this program with testing.txt which is a file with the famous John Quincy Adams letter from his mother.

The output on the terminal is this: Screenshot Output

As you can see in the picture, it just prints out a lot of words that shouldn't even be confused such as 'ages', 'heaven' and 'contest

word nltk python-3.x python
1 Answers

NLTK is designed to help do natural language analysis. I'm really not sure that it's the optimal tool for trying to do spell correction. For one thing, the word list which you are using does not attempt to contain every possible correctly-spelled word, because it assumes that you will use one of the "stemmers" built into NLTK; a stemmer attempts to figure out what each word's "stem" (or base) is. Stemming lets the package analyse "ages" as the plural of "age" and the fact that that will work means that it is unnecessary to include "ages" in the word list.

It is worth noting that NLTK includes utilities which do a much better job of splitting input into words than just calling string.split(), which doesn't know anything about punctuation. If you're going to use NLTK, you'd be well-advised to let it do this work for you, for example with the nltk.word_tokenize function.

Moreover, NLTK generally tries to guess what a word is if it doesn't know it, which means that it will often be able to identify the part of speech of a misspelled or even invented word.

As an example, I ran its default part of speech tagger on Lewis Carroll's famous Jabberwocky, to produce the following output. (I added the definition of each part of speech tag to make it easier to read.) 

>>> poem = """'Twas brillig, and the slithy toves
... did gyre and gimble in the wabe:
... All mimsy were the borogoves,
... and the mome raths outgrabe.
... """
>>> print('\n'.join(f"{word+' :':<12}({tag}) {tagdict[tag][0]}"
...                 for word, tag in nltk.pos_tag(nltk.word_tokenize(poem))))
'T :        (NN) noun, common, singular or mass
was :       (VBD) verb, past tense
brillig :   (NN) noun, common, singular or mass
, :         (,) comma
and :       (CC) conjunction, coordinating
the :       (DT) determiner
slithy :    (JJ) adjective or numeral, ordinal
toves :     (NNS) noun, common, plural
did :       (VBD) verb, past tense
gyre :      (NN) noun, common, singular or mass
and :       (CC) conjunction, coordinating
gimble :    (JJ) adjective or numeral, ordinal
in :        (IN) preposition or conjunction, subordinating
the :       (DT) determiner
wabe :      (NN) noun, common, singular or mass
: :         (:) colon or ellipsis
All :       (DT) determiner
mimsy :     (NNS) noun, common, plural
were :      (VBD) verb, past tense
the :       (DT) determiner
borogoves : (NNS) noun, common, plural
, :         (,) comma
and :       (CC) conjunction, coordinating
the :       (DT) determiner
mome :      (JJ) adjective or numeral, ordinal
raths :     (NNS) noun, common, plural
outgrabe :  (RB) adverb
. :         (.) sentence terminator

NLTK is an extraordinary body of work with a lot of practical applications. I'm just not sure that yours is one of them. But if you haven't already done so, do take a look at the Creative Commons licensed book which describes NLTK, Natural Language Processing with Python. Not only is the book a good guide to the NLTK library, it is also a gentle introduction to text processing in Python 3.

user_15662210
Related