Why don't I get an error if a column referenced in a negative filter doesn't exist, and how might I add the error message?

user_9731173 0
Viewed 36

Take the following two dataframes:

onedf<-data.frame(ID=c("a","b","c"),
                  Count=c(7,80,23))

anotherdf<-data.frame(Car_ID=c("b","c","d"),
                      Color=c("red","green","blue"))

In an effort to get a filtered version of onedf where IDs are only preserved if they don't appear in anotherdf, I mistakenly refer to anotherdr$ID in the code below:

filtereddf<-onedf%>%
  filter(!(ID %in% anotherdf$ID))

Why is there no warning or error message produced, given the fact that no column named ID exists in anotherdf?

The result of running the above is simply filtereddf actually being an unfiltered version of onedf. I think, however, that the code should brick, and alert me that anotherdf$ID does not exist. If this happened, I would know to check my column names within anotherdf and would end up being able to correctly edit my code to

filtereddf<-onedf%>%
  filter(!(ID %in% anotherdf$Car_ID))

ultimately resulting in the filtereddf being a single row,

enter image description here

In practice, this filter is one of many steps in a project, and so the fact that no filtering and no error message is delivered makes this challenging to diagnose as the culprit in the ultimate erroneous output.

Is there a way to achieve this filter in such a way that this misnamed column reference would spark an error, or perhaps a way to force filter() to validate the column name?

dplyr r
1 Answers

The reason it is giving the full row is because if the column doesn't exist, it returns NULL, and when we do a filter with %in% on NULL, it returns FALSE. Negating (!), changes the FALSE to TRUE and thus we get all rows

anotherdf$ID
#NULL
onedf %>% 
    filter(ID %in% NULL)
#[1] ID    Count
#<0 rows> (or 0-length row.names)

onedf %>%
    filter(!ID %in% NULL)
#  ID Count
#1  a     7
#2  b    80
#3  c    23

It can be easily understood, if we create a column with mutate

 onedf %>%
    mutate(flag1 = ID %in% anotherdf$ID, flag2 = !flag1) 
#   ID Count flag1 flag2
#1  a     7 FALSE  TRUE
#2  b    80 FALSE  TRUE
#3  c    23 FALSE  TRUE

We could force to stop if there are no columns with that column name exist in 'anotherdf'

onedf %>%
     filter({stopifnot(exists('ID', where = anotherdf)); ID %in% anotherdf$ID})
Error: Problem with `filter()` input `..1`.
✖ exists("ID", where = anotherdf) is not TRUE
ℹ Input `..1` is `{ ... }`.
Run `rlang::last_error()` to see where the error occurred.
user_37322710
Related