error in regex pattern matching for text retrieval into two columns of a dataframe

user_6179906 0
Viewed 352

Consider the below hypothetical data:

x <- "There is a horror movie running in the iNox theater. : If row names are supplied of length one and the data 
frame has a single row, the row.names is taken to specify the row names and not a column (by name or number). 
If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify 
the row names and not a column (by name or number) Can we go : Please"


y <- "There is a horror movie running in the iNox theater. If row names are supplied of length one and the data 
frame has a single row, the row.names is taken. To specify the row names and not a column. By name or number. : 
If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify 
the row names and not a column (by name or number) Can we go : Please"

z <- "There is a horror movie running in the iNox theater. If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number). 
If row names are supplied of length one. : And the data frame has a single row, the row.names is taken to specify 
the row names and not a column (by name or number) Can we go : Please"

df <- data.frame(Text = c(x, y, z), row.names = NULL, stringsAsFactors = F)

Did you notice that there is an ":" at different locations. For example:

  • In 'x' it ( ":" ) is after the first sentence.
  • In 'y' it ( ":" ) is after the fourth sentence.
  • and In 'z' it is after the sixth sentence.
  • Moreover there is one more ":" before the last sentence in the each text.

What I want to do, create two columns such that:

  • Only the first ":" is considered and NOT THE LAST ONE.
  • If there is a ":" within first three sentences, then divide the whole text into two columns otherwise, keep all the text in the second columns and 'NA' in the first column.

Wanted Output for 'x':

 Col1                                                        Col2 
 There is a horror movie running in the iNox theater.        If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number). If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number) Can we go : Please

Wanted Output for 'y' (because ":" is not found within first three sentences, therefore) :

 Col1     Col2 
 NA       There is a horror movie running in the iNox theater. If row names are supplied of length one and the data frame has a single row, the row.names is taken. To specify the row names and not a column. By name or number. : If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number) Can we go : Please

Just like above result for 'y', the Wanted Output result for 'z' should be:

  Col1    Col2
  NA      all of the text from 'z'

What I am trying to do is:

resX <- data.frame(Col1 = gsub("\\s\\:.*$","\\1", df$Text[[1]]), 
           Col2 = gsub("^[^:]+(?:).\\s","\\1", df$Text[[1]]))

resY <- data.frame(Col1 = gsub("\\s\\:.*$","\\1", df$Text[[2]]), 
           Col2 = gsub("^[^:]+(?:).\\s","\\1", df$Text[[2]]))

resZ <- data.frame(Col1 = gsub("\\s\\:.*$","\\1", df$Text[[3]]), 
           Col2 = gsub("^[^:]+(?:).\\s","\\1", df$Text[[3]]))

And then merging above into a resulting dataframe "resDF" using rbind.

Issues are:

  • The above can be done using "for() loop" or anyother method to make code simpler.
  • The result from "y" and "z" text are not coming as I wanted (shown above).
dataframe r perl regex
4 Answers

You can try with this negative look ahead regex:

^(?s)(?!(?:(?:[^:]*?\\.){3,}))(.*?):(.*)$

Regex Demo and Detailed explanation of the regex

Updated:

If your condition is met then the regex will return true and you should get 2 part

group 1 contains the value until first : and group 2 will contain value after that.

If condition not met then you copy the whole string to column 2 and put whatever you want as column 1

An updated sample snippet which contains a method named process data will do the tricks for you. if the condition is met then it will split the data and put in col1 and col2.... if the condition is not met in case of y and z in your input... it will put NA in the col1 and the entire value in col2.

Run the Sample Source --> ideone:

library(stringr)

    x <- "There is a horror movie running in the iNox theater. : If row names are supplied of length one and the data 
    frame has a single row, the row.names is taken to specify the row names and not a column (by name or number). 
    If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify 
    the row names and not a column (by name or number) Can we go : Please"


    y <- "There is a horror movie running in the iNox theater. If row names are supplied of length one and the data 
    frame has a single row, the row.names is taken. To specify the row names and not a column. By name or number. : 
    If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify 
    the row names and not a column (by name or number) Can we go : Please"

    z <- "There is a horror movie running in the iNox theater. If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number). 
    If row names are supplied of length one. : And the data frame has a single row, the row.names is taken to specify 
    the row names and not a column (by name or number) Can we go : Please"             


df <- data.frame(Text = c(x, y, z), row.names = NULL, stringsAsFactors = F)

resDF <- data.frame("Col1" = character(), "Col2" = character(), stringsAsFactors=FALSE)

   processData <- function(a) {
        patt <- "^(?s)(?!(?:(?:[^:]*?\\.){3,}))(.*?):(.*)$"    
        if(grepl(patt,a,perl=TRUE))
        {
            result<-str_match(a,patt)    
            col1<-result[2]
            col2<-result[3]
        }
        else
        {
            col1<-"NA"
            col2<-a
        }
       return(c(col1,col2))

    }



for (i in 1:nrow(df)){
tmp <- df[i, ]
resDF[nrow(resDF) + 1, ] <- processData(tmp)
}    


print(resDF)

Sample Output:

                                                   Col1
1 There is a horror movie running in the iNox theater. 
2                                                    NA
3                                                    NA
                                                                                                                                                                                                                                                                                                                                                                                                                              Col2
1                                                        If row names are supplied of length one and the data \n    frame has a single row, the row.names is taken to specify the row names and not a column (by name or number). \n    If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify \n    the row names and not a column (by name or number) Can we go : Please
2 There is a horror movie running in the iNox theater. If row names are supplied of length one and the data \n    frame has a single row, the row.names is taken. To specify the row names and not a column. By name or number. : \n    If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify \n    the row names and not a column (by name or number) Can we go : Please
3      There is a horror movie running in the iNox theater. If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number). \n    If row names are supplied of length one. : And the data frame has a single row, the row.names is taken to specify \n    the row names and not a column (by name or number) Can we go : Please
user_7611320

Brief

I was inspired by Rizwan's answer to make mine, so you'll see his answer completes mine. What I didn't like was the fact that it breaks on non-sentence starts (such as row.names - although the text samples the OP provided don't provide any examples where row.names is present 3 times in the first 2 sentences to showcase this). I also ensured the capture groups/columns are numbered exactly as the OP expects and that there is always a match. My answer is really an improvement of Rizwan's.

Note 1: I am assuming a "sentence" is defined by a period/dot, followed by at least one horizontal space .

Note 2: This works with PCRE regex and is untested with other regex flavours and may need to be adapted to other regex flavours to work properly (namely the if/else, vertical whitespace, and horizontal whitespace tokens)

Code

See this code in use here

^(?(?!(?:[^:\v]*?\.\h){3,})([^:\v]*?)\s*:\s*|)(.*)$

Results

Input

There is a horror movie running in the iNox theater. : If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number). If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number) Can we go : Please

There is a horror movie running in the iNox theater. If row names are supplied of length one and the data frame has a single row, the row.names is taken. To specify the row names and not a column. By name or number. : If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number) Can we go : Please

There is a horror movie running in the iNox theater. If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number). If row names are supplied of length one. : And the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number) Can we go : Please

Output

Match 1

  • Group 1: There is a horror movie running in the iNox theater.
  • Group 2: If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number). If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number) Can we go : Please

Match 2

  • Group 1: empty - no match
  • Group 2: There is a horror movie running in the iNox theater. If row names are supplied of length one and the data frame has a single row, the row.names is taken. To specify the row names and not a column. By name or number. : If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number) Can we go : Please

Match 3

  • Group 1: empty - no match
  • Group 2: There is a horror movie running in the iNox theater. If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number). If row names are supplied of length one. : And the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number) Can we go : Please

Explanation

  • ^ Assert position at the beginning of the string
  • (?(?!(?:[^:\v]*?\.\h){3,})([^:\v]*?)\s*:\s*|)
    • (?(?!...)x|y) If statement using negation (?!...) as condition
      • (?:[^:\v]*?\.\h){3,} Match the following at least 3 times
      • [^:\v]*? Match any character not present in the set (not colon or vertical whitespace characters) any number of times, but as few as possible
      • \.\h Match a dot character literally, followed by a horizontal whitespace character (space or tab)
      • If statement true: If the above is met, do the following
      • ([^:\v]*?)\s*:\s*
        • ([^:\v]*?) Capture into group 1: Any character not present in the set (not colon or vertical whitespace characters) any number of times, but as few as possible
        • \s*:\s* Match any number of whitespace characters, followed by a colon, followed by any number of whitespaces (note you can change * to + if there is always at least 1 whitespace character trailing/leading the colon improve matchmaking in cases where a "sentence" might contain a :)
      • If statement false: The previous conditions were not met, do the following: Match nothing
  • (.*) Capture into group 2: Any character (excludes newline characters when s flag is off) any number of times
  • $ Assert position at the end of the string
user_36007090

Negative lookahead is expensive and very hard to read. Here's a much simpler solution:

library(stringr)

# throw out everything after first :, and count the number of sentences
split = str_count(sub(':.*', '', df$Text), fixed('. ')) < 3

# assemble the required data (you could also avoid ifelse if really needed)
data.frame(col1 = ifelse(split, sub(':.*', '', df$Text), NA),
           col2 = ifelse(split, sub('.*?:', '', df$Text), df$Text))
user_8177780

Split into sentences; grep for where : first occurs, and use a conditional to split the original text:

sp <- strsplit(x, '(?<=\\.)(?=\\s+\\S)', perl = TRUE)[[1L]]
sp <- if (grep(':', sp)[1L] < 3L)
  sub(':\\s+', '$', x) else paste0('$', x)
sp <- gsub('\\v', '', sp, perl = TRUE)

str(read.table(text = sp, sep = '$', col.names = paste0('Col', 1:2), as.is = TRUE))

# 'data.frame': 1 obs. of  2 variables:
#   $ Col1: chr "There is a horror movie running in the iNox theater. "
#   $ Col2: chr "If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names an"| __truncated__

Write a handy function to make your work easier:

For example, you can use different punctuation to signal the end of a sentence (eg, end_of_sentence = '.!?)' will split text into sentences if one of .!?) is followed by whitespace); n allows you to control the number of sentences to look for the first :; and you can change sep if you expect $ in your text (choose a character here that likely won't be in your text)

f <- function(text, end_of_sentence = '.', n = 3L, sep = '$') {
  p <- sprintf('(?<=[%s])(?=\\s+\\S)', end_of_sentence)

  sp <- strsplit(text, p, perl = TRUE)[[1L]]
  sp <- if (grep(':', sp)[1L] <= n)
    sub(':\\s+', sep, text) else paste0(sep, text)
  sp <- trimws(gsub('\\v', '', sp, perl = TRUE))

  read.table(text = sp, sep = sep, col.names = paste0('Col', 1:2),
             stringsAsFactors = FALSE)
}

## test
f(x); f(y); f(z)

## vectorize it to work on more than one string
f <- Vectorize(f, SIMPLIFY = FALSE, USE.NAMES = FALSE)

do.call('rbind', f(df$Text))

#   Col1
# 1 There is a horror movie running in the iNox theater. 
# 2                                                  <NA>
# 3                                                  <NA>
#   Col2
# 1 If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number). If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number) Can we go : Please
# 2 There is a horror movie running in the iNox theater. If row names are supplied of length one and the data frame has a single row, the row.names is taken. To specify the row names and not a column. By name or number. : If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number) Can we go : Please
# 3 There is a horror movie running in the iNox theater. If row names are supplied of length one and the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number). If row names are supplied of length one. : And the data frame has a single row, the row.names is taken to specify the row names and not a column (by name or number) Can we go : Please
user_29949490
Related