Filter values relative to values in another column using dplyr

user_1553411 0
Viewed 279

I have a column in a dataframe and I would like to filter out any rows that are over or under two standard deviations from the mean.

As an example, I would hope to get two rows out of this (only the rows that fall between the low and high standard deviations: 

group    value    low_sd    high_sd    
a          4        2          8
a          1        2          8
b          6        4          9
b          12       4          9

I was hoping to use dplyr::between . 

clean_df <- df%>%
  filter(between(value, low_sd, high_sd))

But it seems between only takes numerical values.

dplyr r
2 Answers

The between is not vectorized for left, right values. Instead, this can be done by using only the comparison (>/<) operators

library(dplyr)
df %>% 
    filter(value > low_sd, value <= high_sd)
#    group value low_sd high_sd
#1     a     4      2       8
#2     b     6      4       9

But if we wrap with Vectorize, it would work as well

df %>% 
   filter(Vectorize(dplyr::between)(value, low_sd, high_sd))
#    group value low_sd high_sd
#1     a     4      2       8
#2     b     6      4       9

data

df <- structure(list(group = c("a", "a", "b", "b"), value = c(4L, 1L, 
6L, 12L), low_sd = c(2L, 2L, 4L, 4L), high_sd = c(8L, 8L, 9L, 
9L)), class = "data.frame", row.names = c(NA, -4L))
user_37322710

Alternatively, you can use between() from data.table:

df %>%
 filter(data.table::between(value, low_sd, high_sd))

  group value low_sd high_sd
1     a     4      2       8
2     b     6      4       9

Or if you want to stick just to dplyr:

df %>%
 rowwise() %>%
 filter(dplyr::between(value, low_sd, high_sd))
user_59645570
Related