Is there an algorithm for removing a dash ("-") between two words and then contracting them?

user_15367665 0
Viewed 217

I have a lot of text of words with dashes between new lines like so:

vec <- "Today is a good day because the sun is shin- ing."

What I want is instead:

"Today is a good day because the sun is shining."

But I don't want it just for specific words but for all words that are being "broken up" like that. It seems like something you should be able to do in Word format, but I haven't been able to figure out how, so maybe it's more complicated.

For the record, I am using readtext/quanteda package, but I can't find anything there either that can do this by default at least.

Is there some simple way to do this?

quanteda topic-modeling r text
2 Answers

It's certainly a good solution to fix this in the character inputs before creating quanteda objects (corpus, tokens, etc.). An alternative in quanteda is to tokenise the texts with the ending hyphens, then:

  • compound the hyphenated tokens with the token that follows
  • remove the new tokens with their internal hyphens

Example:

library("quanteda")
## Package version: 3.0.0
## Unicode version: 10.0
## ICU version: 61.1
## Parallel computing: 12 of 12 threads used.
## See https://quanteda.io for tutorials and examples.

txt <- c(
  "The sun is shin- ing.",
  "Hyphen- ation is fun",
  "text an- alysis"
)
toks <- tokens(txt)
toks
## Tokens consisting of 3 documents.
## text1 :
## [1] "The"   "sun"   "is"    "shin-" "ing"   "."    
## 
## text2 :
## [1] "Hyphen-" "ation"   "is"      "fun"    
## 
## text3 :
## [1] "text"   "an-"    "alysis"

The compounding step:

toksc <- tokens_compound(toks, phrase("*- *"), concatenator = "")
toksc
## Tokens consisting of 3 documents.
## text1 :
## [1] "The"      "sun"      "is"       "shin-ing" "."       
## 
## text2 :
## [1] "Hyphen-ation" "is"           "fun"         
## 
## text3 :
## [1] "text"      "an-alysis"

And finally the replacement without hyphens step:

toks_hyphenated <- grep("\\w+-\\w+", types(toksc), value = TRUE)
tokens_replace(toksc, toks_hyphenated, gsub("-", "", toks_hyphenated))
## Tokens consisting of 3 documents.
## text1 :
## [1] "The"     "sun"     "is"      "shining" "."      
## 
## text2 :
## [1] "Hyphenation" "is"          "fun"        
## 
## text3 :
## [1] "text"     "analysis"

Edit: Added to question

If you really want to recombine these to make a corpus from the processed tokens, you can apply this step:

> toks_rejoined <- tokens_replace(toksc, toks_hyphenated, gsub("-", "", 
> corpus(sapply(toks_rejoined, paste, collapse = " "))
Corpus consisting of 3 documents.
text1 :
"The sun is shining ."

text2 :
"Hyphenation is fun"

text3 :
"text analysis"
user_41582740

Here is one way. We can use str_replace_all from the stringr package.

vec <- "Today is a good day because the sun is shin- ing."

library(stringr)

vec2 <- str_replace_all(vec, "-\\s+", "")

vec2
# [1] "Today is a good day because the sun is shining."
user_76698090
Related