pyspellchecker: do not split URL

user_3191371 0
Viewed 200

I tried to set-up an autocorrect using pyspellchecker in Python. In general, it does work, however it currently also splits the URLs, which is not really desired. The code is as following:

from spellchecker import SpellChecker

spell = SpellChecker()
words = spell.split_words("This is my URL https://test.com")
test = [spell.correction(word) for word in words]

This result in the following: ['This', 'is', 'my', 'URL', 'steps', 'test', 'com']

What do I have to change that all URLs are not autocorrected?

autocorrect spell-checking python
3 Answers

If you use the base str.split to split the sentence into words at each space it will work (you will have lost functionality in splitting words separated by anything other than spaces)

from spellchecker import SpellChecker

spell = SpellChecker()
words = str.split("This is my URL https://test.com")
test = [spell.correction(word) for word in words]

Output:

['This', 'is', 'my', 'usl', 'https://test.com']

user_111553820

You can define your own tokenizer that you pass to the SpellChecker class so that it will only split on whitespace (or anything else you want):

from spellchecker import SpellChecker

def splitter(words):
    return words.split(" ")    # split on whitespace

spell = SpellChecker(tokenizer=splitter)
words = spell.split_words("This is my URL https://test.com")
test = [spell.correction(word) for word in words]

EDIT: FYI, the reason it behaves this way is because it looks like the default tokenizer uses this regex to split the text into words.

user_85395330

NLTK's TweetTokenizer correctly tokenizes URLs, hashtags, and emoticons.

>>> from nltk.tokenize import TweetTokenizer
>>> tknzr = TweetTokenizer()
>>> tknzr.tokenize(s)
['This', 'is', 'my', 'URL', 'https://test.com']

NLTK comes with a variety of state-of-the-art word tokenization primitives. I suggest you use NLTK to turn your string into words before filtering for autocorrection. You can use NLTK's part-of-speech utilities to determine what things should be autocorrected.

user_14596690
Related