Filter samples by multiple columns in a stepwise way then add each step to the same dataframe

user_12552636 0
Viewed 38

I am trying to filter a large dataset in a hierarchical manner. So for example I want to filter df by filter A, then the newly filtered df by B, then C and so on.

Along the way however I want to add the samples that are removed from the df by each filter to a master_df that contains the samples that are removed and the corresponding filter as the column name.

I have tried the following for the below df:

sample    speed   time   country
1         10      100    UK
2         20      100    UK
3         1       300    USA
4         35      500    UK
5         55      400    UK
6         20      300    UK

df2 <- filter(df, speed > 10)
df3 <- filter(df2, time > 100, nHet < 400)

removed <- df %>%
    select(sample, speed, time) %>%
    filter(speed <= 10)%>%
    filter(
    between(time < 100, time > 400))

The issue I am finding is that I am unable to perform this all coherently in one go, that is to generate the list with the samples removed and the separate list with the samples I have removed in it, based on what the filter was. Finally (thanks for baring with me) although I seem to be able to filter the samples based on those samples inside of the range time when creating df3, I am unable to do the reverse when using dplyr to get those that are outside that range.

My desired output would be a dataframe removed that looks like this.

sample    speed   time   
1         10                 
3         1              
4                 500

Any help would be greatly appreciated! Thank you!

dplyr r
1 Answers

What you have currently is & condition, you need to change the condition as well as the operator so the condition should be separated by | :

library(dplyr)
df %>% filter(speed <= 10 | time < 100 | time > 400)

#  sample speed time country
#1      1    10  100      UK
#2      3     1  300     USA
#3      4    35  500      UK

If you want to "remove" those values, we can turn them to NA

df %>% 
  filter(speed <= 10 | time < 100 | time > 400) %>%
  mutate(speed = replace(speed, speed > 10, NA), 
         time = replace(time, time > 100 | time > 400, NA))

#  sample speed time country
#1      1    10  100      UK
#2      3     1   NA     USA
#3      4    NA   NA      UK
user_39629140
Related