How to choose the longest ngram in a repetitive string in R?

user_11304830 0
Viewed 32

I have a dataset that looks like the following one (just with more rows):

x = c("abov level", "abov level consist", "abov level consist price", 
"abov level consist price stabil", "abov level consist price stabil protract", 
"abov level consist price stabil protract period", "abov level consist price stabil protract period time", 
"abov level consist price stabil sinc", "abov level consist price stabil sinc last", 
"abov level consist price stabil sinc last autumn", "abov level consist price stabil some", 
"abov level consist price stabil some time", "abov over", "abov over come", 
"abov over come month", "abov precis", "abov precis level", "abov precis level depend", 
"abov precis level depend futur", "abov precis level depend futur energi", 
"abov precis level depend futur energi price", "abov precis level depend futur energi price develop"
)

 [1] "abov level"                                          
 [2] "abov level consist"                                  
 [3] "abov level consist price"                            
 [4] "abov level consist price stabil"                     
 [5] "abov level consist price stabil protract"            
 [6] "abov level consist price stabil protract period"     
 [7] "abov level consist price stabil protract period time"
 [8] "abov level consist price stabil sinc"                
 [9] "abov level consist price stabil sinc last"           
[10] "abov level consist price stabil sinc last autumn"    
[11] "abov level consist price stabil some"                
[12] "abov level consist price stabil some time"           
[13] "abov over"                                           
[14] "abov over come"                                      
[15] "abov over come month"                                
[16] "abov precis"                                         
[17] "abov precis level"                                   
[18] "abov precis level depend"                            
[19] "abov precis level depend futur"                      
[20] "abov precis level depend futur energi"               
[21] "abov precis level depend futur energi price"         
[22] "abov precis level depend futur energi price develop"

As you can see there is a clear pattern: one word at a time is added to the previous ngram before changing base and restarting the process again. Let me take the first "block" as an example:

 [1] "abov level"                                          
 [2] "abov level consist"                                  
 [3] "abov level consist price"                            
 [4] "abov level consist price stabil"                     
 [5] "abov level consist price stabil protract"            
 [6] "abov level consist price stabil protract period"     
 [7] "abov level consist price stabil protract period time"

For every "block" like the one above, I would only keep the longest sentence/ngram. In the above case I would only keep row number seven. Doing this for every block, I would get:

    
 [1] "abov level consist price stabil protract period time"           
 [2] "abov level consist price stabil sinc last autumn"    
 [3] "abov level consist price stabil some time"                                              
 [4] "abov over come month"                                      
 [5] "abov precis level depend futur energi price develop"

Can anyone help me do that?

Thanks!

gsub dataframe substring r string
2 Answers

We can use filter in dplyr with lead

library(dplyr)
tibble(x) %>%
     filter((nchar(lead(x, default = last(x))) - nchar(x)) <= 0)
user_37322710

You can count the number of characters in each string and select the values where the characters are less than the previous string.

inds <- c(which(diff(nchar(x)) < 0), length(x))
x[inds]

#[1] "abov level consist price stabil protract period time"
#[2] "abov level consist price stabil sinc last autumn"    
#[3] "abov level consist price stabil some time"           
#[4] "abov over come month"                                
#[5] "abov precis level depend futur energi price develop"
user_39629140
Related