Fastest way to check occurrence of a set of substrings in a large collection of documents using R

user_4883787 0
Viewed 53

I have a large collection of documents, dc, (with several million rows) with the following data.frame structure

doc_id    body
  1       'sdfadfs...'
  2       'dfadf...'
  3       'sadf....'

I also have about 10,000 terms (or substrings) stored in terms (terms=c('sfa','adfa','dfad',...)).

I want to find the occurrence of each term for each document of dc. In the end, I want the result to be something like this

doc_id term
  1    'sfa'
  1    'dfad'
  2    'adfa'
  3    'sfa'
  3    'dfad'

Currently, I'm using the following code (with the help of stri_detect_fixed)

res_all=lapply(terms,function(term_i){ #loop over each term
    res=stri_detect_fixed(dc$body,term_i) #check occurrence of one term in each document
    
    data.frame(doc_id=which(res),term=term_i)
   
  })
  
bind_rows(res_all)

However, the above code is quite slow. Is there anything I can do to speed up the code?

stringi substring r string performance
1 Answers

Collapse the term values in one pattern and use it in stringr::str_extract_all which will return a list. Use unnest to get each term as separate row.

library(dplyr)

terms=c('sfa','adfa','dfad', ...)
pat <- sprintf('\\b(%s)\\b', paste0(terms, collapse = '|'))

df %>%
  mutate(term = stringr::str_extract_all(body, pat)) %>%
  tidyr::unnest(term)

Word boundaries (\\b) are added to the pattern so that 'sfa' does not match with terms like 'asfa' or 'sfad' etc.

user_39629140
Related