Remove repeating characters from sentence but retain the words meaning

Question

I want to remove repeating characters from a sentence but make it so that the words still retain its meaning (if it has any). For example : I'm so haaappppyyyy about offline school to I'm so happy about offline school. See, haaappppyyyy became happy and offline & school stay the same instead becoming ofline & schol

I've tried two solutions, using RE and itertools, but none really fits for what I'm searching for

Using Regex :

tweet = 'I'm so haaappppyyyy about offline school'
repeat_char = re.compile(r"(.)\1{1,}", re.IGNORECASE)
tweet = repeat_char.sub(r"\1\1", tweet)
tweet = re.sub("(.)\1{2,}", "\1", tweet)

output :

I'm so haappyy about offline school #it makes 2 chars for every repating chars

using itertools :

tweet = 'I'm so happy about offline school'
tweet = ''.join(ch for ch, _ in itertools.groupby(tweet))

output :

I'm so hapy about ofline schol

How can I fix this? should I make a lists of words I want to exclude?

In addition, I want it to also be able to reduce some words that's in a pattern to it's base form. For example :

wkwk (base form)
wkwkwkwk
wkwkwkwkwkwkwk

I want to make the second and the third word into the first word, the base form

user_3832970 · Answer

You can combine regex and NLP here by iterating over all words in a string, and once you find one with identical consecutive letters reduce them to max 2 consecutive occurrences of the same letters and run the automatic spellcheck to fix the spelling.

See an example Python code:

import re
from textblob import TextBlob
from textblob import Word
rx = re.compile(r'([^\W\d_])\1{2,}')
print( re.sub(r'[^\W\d_]+', lambda x: Word(rx.sub(r'\1\1', x.group())).correct() if rx.search(x.group()) else x.group(), tweet) )
# => "I'm so happy about offline school"

The code uses the Textblob library, but you may use any you like.

Note that ([^\W\d_])\1{2,} matches any three or more consecutive letters, [^\W\d_]+ matches one or more letters.

user_5730279 · Answer

This answer was originally written for Regex to reduce repeated chars in a string which was closed as duplicate before I could submit my post. So I "recycled" it here.

Regex is not always the best solution

Regex for validation of formats or input

A regex is often used for low-level pattern recognition and substitution. It may be useful for validation of formats. You can see it as "dump" automation.

Linguistics (NLP)

When it comes to natural language (NLP), or here spelling (dictionary) the semantics may play a role. Depending on the context "ass" and "as" may both be correctly spelled, although the semantics are very different. (I apologize for the rude examples, but I am not a native-speaker and those two had the most distinct meaning depending on re-duplication).

For those cases a regex or simple pattern-recognition may be not sufficient. It can cause more effort to apply it correctly than the research for a language-specific library or solution (including a basic application).

Examples for spelling that a regex may struggle with

Like the difference between "haappy" (orthographically invalid, but only the duplicated vowels "aa", not the consonants "pp") and "yeees" (contains no duplicates in correct spelling) or "kiss" (is correctly spelled with duplicate consonants)

Spelling correction requires more

For example a dictionary to lookup if duplicate characters (vowels or consonants) are valid for correct spelling of the word in its form.

Consider a spelling-correction module

You could use textblob module for spelling correction:

To install: pip install textblob

Example for some test-cases (independent words):

from textblob import TextBlob
 
incorrect_words = ["cmputr", "yeees", "haappy"]  # incorrect spelling

text = ",".join(incorrect_words)  # join them as comma separated list
print(f"original words: {text}")
 
b = TextBlob(text)

# prints the corrected spelling
print(f"corrected words: {b.correct()}")

Prints:

original words: cmputr,yeees,haappy
corrected words: computer,eyes,happy

Surprise: You might have expected "yes" (so did I). But the correction results not in removal of 2 duplicated vowels "ee", but rearrangement to keep almost all letters (5 of 6, only removed one "e").

Example for the given sentence:

from textblob import TextBlob

tweet = "I'm so haaappppyyyy about offline school"  # either escape or use different quotes when a single-quote (') is enclosed
print(TextBlob(tweet).correct())