Meaningless Spacy Nouns

user_2273230 0
Viewed 325

I am using Spacy for extracting nouns from sentences. These sentences are grammatically poor and may contain some spelling mistakes as well.

Here is the code that I am using:

Code

import spacy
import re

nlp = spacy.load("en_core_web_sm")

sentence= "HANDBRAKE - slow and fast (SFX)"
string= sentence.lower()
cleanString = re.sub('\W+',' ', string )
cleanString=cleanString.replace("_", " ")

doc= nlp(cleanString)

for token in doc:
    if token.pos_=="NOUN":
        print (token.text)

Output:

sfx

Similarly for sentence "fast foward2", I get Spacy noun as

foward2

Which shows that these nouns have some meaningless words like: sfx, foward2, ms, 64x, bit, pwm, r, brailledisplayfastmovement, etc.

I only want to keep phrases that contain sensible single-word nouns like broom, ticker, pool, highway etc.

I have tried Wordnet to filter common nouns between wordnet and spacy but it is a bit strict and filter some sensible nouns as well. For example, it filters nouns like motorbike, whoosh, trolley, metal, suitcase, zip etc

Therefore, I am looking for a solution in which I can filter out most sensible nouns from spacy nouns list that I have obtained.

spacy wordnet text python
2 Answers

It seems you can use pyenchant library:

Enchant is used to check the spelling of words and suggest corrections for words that are miss-spelled. It can use many popular spellchecking packages to perform this task, including ispell, aspell and MySpell. It is quite flexible at handling multiple dictionaries and multiple languages.

More information is available on the Enchant website:

https://abiword.github.io/enchant/

Sample Python code:

import spacy, re
import enchant                        #pip install pyenchant

d = enchant.Dict("en_US")
nlp = spacy.load("en_core_web_sm")

sentence = "For example, it filters nouns like motorbike, whoosh, trolley, metal, suitcase, zip etc"
cleanString = re.sub('[\W_]+',' ', sentence.lower()) # Merging \W and _ into one regex

doc= nlp(cleanString)
for token in doc:
    if token.pos_=="NOUN" and d.check(token.text):
        print (token.text)
# => [example, nouns, motorbike, whoosh, trolley, metal, suitcase, zip]
user_38329700

While using pyenchant spellchecker, I have found it useful to do the check after converting the word to uppercase fully. Also Splitting the sentence/phrase and feeding the words one at a time gives better results.

Example:

enchantChecker.check("landsat".upper()) and enchantChecker.check("wwii".upper()) returns True where as using lowercase words returns False.

If you need to mix more than one spellcheckers, another good one is to check spaCy library's is_oov (out of vocabulary) flag after loading en_core_web_lg

user_46488730
Related