Create binary yes/no animal variable based on match with any term in a dictionary, "animal" in R

user_14307526 0
Viewed 64

Continuing off this question: R: Create category column reflecting match between a dictionary and column in df I have a big dataset, "df", of 30,000 rows, and two big dictionary dataframes: (1) animal, 600k rows; (2)nature, 300k rows.

I am simply trying to figure out how to create two simple binary variables, "df$content_animal" and "df$content_nature" based on whether each row in df$content had any matches with "animal" or "nature" dictionaries. (1=match, 0=no match).

Below are the data samples, it's impossible for me to include the entire datasets here:

df <- tibble(content= c("hello turkey feet blah blah blah", "i love rabbits haha", "wow this sunlight is amazing", "omg did u see the rainbow?!", "turtles like swimming in the water", "i love running across grassy lawns with my dog"))

animal=c("turkey", "rabbit", "turtle", "dog", "cat", "bear")
nature=c("sunlight", "water", "rainbow", "grass", "lawn", "mountain", "ice")

I have tried the following codes based on multiple-pattern matches, to no success - I suspect it is bc of the largeness of both my dataset and dictionary/pattern:

df$content_animal <- grepl(paste(animal,collapse="|"),df$content,ignore.case=TRUE)
df$content_nature <- grepl(paste(nature,collapse="|"),df$content,ignore.case=TRUE)

which returns the error:

Error in grepl(paste(animal,collapse="|"), df$content,  : 
  invalid regular expression, reason 'Out of memory' Error in grepl(paste(nature,collapse="|"), df$content,  : 
  invalid regular expression, reason 'Out of memory'

I also tried:

df<-df %>%
  mutate(
    content_animal = case_when(grepl(animal, content) ~ "1")
  )
df<-df %>%
  mutate(
    content_nature = case_when(grepl(nature, content) ~ "1")
  )

which returns the error:

Problem with `mutate()` input `content_animal`.
ℹ argument 'pattern' has length > 1 and only the first element will be used
ℹ Input `content_animal` is `case_when(grepl(animal, content) ~ "1")`.argument 'pattern' has length > 1 and only the first element will be used
Problem with `mutate()` input `content_nature`.
ℹ argument 'pattern' has length > 1 and only the first element will be used
ℹ Input `content_nature` is `case_when(grepl(nature, content) ~ "1")`.argument 'pattern' has length > 1 and only the first element will be used

I ALSO tried

bench::mark(basic = mutate(df, content_animal = 1L*map_lgl(content, ~any(str_detect(.x, animal))),
                           content_nature = 1L*map_lgl(content, ~any(str_detect(.x, nature)))),
            fixed = mutate(df, content_animal = 1L*map_lgl(content, ~any(str_detect(.x, fixed(animal)))),
                           content_nature = 1L*map_lgl(content, ~any(str_detect(.x, fixed(nature))))))

which ran for over two hours, without giving me any output.

I'm really at a loss here as to what I should do. Does anyone have any ideas? It there a better package or code to use for my big data purposes???

nlp r
2 Answers

It may be better to loop with lapply and Reduce

Reduce(`|`, lapply(nature, function(x) grepl(x, df$content, ignore.case = TRUE)))
#[1] FALSE FALSE  TRUE  TRUE  TRUE  TRUE

which is the same as

grepl(paste(nature,collapse="|"),df$content,ignore.case=TRUE)
#[1] FALSE FALSE  TRUE  TRUE  TRUE  TRUE
user_37322710

Here's an approach with the quanteda package, which has built-in functions for doing exactly what you want. (I tried this only on the sample dataset; I'd be interested to hear what its performance is on the whole thing.)

library(quanteda)
c = corpus(df$content)
d = dictionary(list(animal = animal, nature = nature))
df = cbind(df, convert(dfm(c, dictionary = d), to = "data.frame")[,-1])
user_108988750
Related