Extract sentences from texts in data frame

user_16185837 0
Viewed 161

I have a data frame with a column "text" and in each row of my data frame "text" contains several sentences (maybe only two, maybe 100 or more). Now I would like to analyze the text in every row of my data frame for specific keywords. If a keyword is found in the text of this row I would like to extract the sentences, which contain keywords, to a separate column, f.e.

needles = c("first", "hope", "analyze", "happy")

mydata <- data.frame(
  text = c("This is the first sentence. It is the beginning of this project",
           "My second sentence is this. I hope this project will work fine. Then I will analyze the third sentence.",
           "And this is the last sentence. Finally my work ends. I am really happy about that.",
           "These sentences do not contain any relevant information. There is no keyword. And it is not relevant."),
  findings = c("This is the first sentence.",
               "I hope this project will work fine. Then I will analyze the third sentence.",
               "I am really happy about that.",
               NA)
)

So column "text" contains the sentences I want to check for keywords, "findings" is the result I would like to have in the end.

Can anyone help me how to apply the solution for all rows of the data frame? Thank you!

r
4 Answers

What about something like this:

find_sentence <- function(text, word){
  require(stringr)
  x <- c(str_split(text, "\\..", simplify=TRUE))
  inds <- which(str_detect(x, word))
  if(length(inds) > 0){
    list(x[inds])
  }else{
    list(NA)
  }
  
}

mydata %>% 
  rowwise %>% 
  mutate(res = find_sentence(text, "the")) %>% 
  unnest(res)

# # A tibble: 4 × 3
#   text                                                                                                    findings                     res            
#   <chr>                                                                                                   <chr>                        <chr>          
# 1 This is the first sentence. It is the beginning of this project                                         This is the first sentence.  This is the fi…
# 2 This is the first sentence. It is the beginning of this project                                         This is the first sentence.  It is the begi…
# 3 My second sentence is this. I hope this project will work fine. Then I will analyze the third sentence. I hope this project will wo… Then I will an…
# 4 And this is the last sentence. Finally my work ends. I am really happy about that.                      I am really happy about tha… And this is th…

This returns a new variable called res that has a different row for each occurrence of the keyword in a sentence. So, if two sentences contained the word (as in the first sentence in text), the text and findings columns will be replicated for each of the relevant sentences in res.

user_82064340

With Base R,

lookup <- strsplit(as.character(mydata[,1]),"\\.")

out <- lapply(lookup,function(x) { 
                logic <- grepl(paste0(needles,collapse="|"),x)
                paste0(x[logic],collapse=".")


            })


data.frame(findings = do.call(rbind,out) )

gives,

#                                                                     findings
#1                                                  This is the first sentence
#2  I hope this project will work fine. Then I will analyze the third sentence
#3                                                I am really happy about that
#4
user_72243540

This uses grep and a strsplit to get the matches.

mydata$findings <- sapply( strsplit( t(mydata), "\\. " ), function(x)
                     x[unlist( lapply( needles, function(y) grep(y, x) ) )] )

                                                                                                     text
1                                         This is the first sentence. It is the beginning of this project
2 My second sentence is this. I hope this project will work fine. Then I will analyze the third sentence.
3                      And this is the last sentence. Finally my work ends. I am really happy about that.
4   These sentences do not contain any relevant information. There is no keyword. And it is not relevant.
                                                                     findings
1                                                  This is the first sentence
2 I hope this project will work fine, Then I will analyze the third sentence.
3                                               I am really happy about that.
4
user_94620950

We can work with list columns by splitting each row in different sentences and look for the needles inside each resulting sentence of each row.

The reduce functions are to take levels of depth of the lists.

code:

library(tidyverse)


needles <- c("first", "hope", "analyze", "happy")

mydata <- data.frame(
  text = c(
    "This is the first sentence. It is the beginning of this project",
    "My second sentence is this. I hope this project will work fine. Then I will analyze the third sentence.",
    "And this is the last sentence. Finally my work ends. I am really happy about that.",
    "These sentences do not contain any relevant information. There is no keyword. And it is not relevant."
  ),
  findings = c(
    "This is the first sentence.",
    "I hope this project will work fine. Then I will analyze the third sentence.",
    "I am really happy about that.",
    NA
  )
)



df <- as_tibble(mydata) %>% 
mutate(mydata, findings = str_split(text, "\\.\\s") %>%
         map(~str_subset(., rebus::or1(needles))) %>% 
         map_if(~length(.) > 1, ~reduce(., ~paste(.x, .y, sep = '. '))),
       findings = map_if(findings, ~length(.) == 0, ~NA) %>% reduce(c))


df
#> # A tibble: 4 × 2
#>   text                                    findings                              
#>   <chr>                                   <chr>                                 
#> 1 This is the first sentence. It is the … This is the first sentence            
#> 2 My second sentence is this. I hope thi… I hope this project will work fine. T…
#> 3 And this is the last sentence. Finally… I am really happy about that.         
#> 4 These sentences do not contain any rel… <NA>

Created on 2021-11-27 by the reprex package (v2.0.1)

user_111687660
Related