Filtering matrix in R by values across columns

user_9684184 0
Viewed 149

I am wondering if there is a way to quickly and efficiently filter matrices without having to convert them into a data frame or data.table.

Please consider the following matrix:

# Choose control variables 
control_vars <- c("x1", "F_5", "F_7")
my_args = lapply(control_vars, function(x) c("", x))
# Create a matrix of all combinations of the control variables
all_combs = as.matrix(do.call("expand.grid", my_args))

The output looks like this:

     Var1 Var2  Var3 
[1,] ""   ""    ""   
[2,] "x1" ""    ""   
[3,] ""   "F_5" ""   
[4,] "x1" "F_5" ""   
[5,] ""   ""    "F_7"
[6,] "x1" ""    "F_7"
[7,] ""   "F_5" "F_7"
[8,] "x1" "F_5" "F_7"

I want to eliminate any row in this matrix where both F_5 and F_7 are present. This would be row 7 and 8.

For some context, I am trying to run stepwise regressions and want to eliminate rows where there are "redundant" fixed effects. That is, I only need F_5 or F_7, but not both. For example, suppose that F_5 and F_7 that categorize countries into 5 and 7 groups. The latter is more granular than the former, but I don't want them both present in my regression.

This would be easy to do after converting it into a data.frame by counting the number of "F"s in a row and then deleting them, but I am wondering if there is a faster and cleaner solution while using matrices. My real dataset has 8 different substitutable fixed effects, so manually deleting them is also not possible.

Thanks!

dplyr r
2 Answers

Create a logical vector with rowSums on 'F_5' and 'F_7' separately with ==, create a single expression with logical operator & on the compound logical expressions, negate (!) and use that as row index to subset the rows

i1 <- !(rowSums(all_combs == 'F_5') > 0 & rowSums(all_combs == 'F_7') > 0)
all_combs[i1, ]
#   Var1 Var2  Var3 
#[1,] ""   ""    ""   
#[2,] "x1" ""    ""   
#[3,] ""   "F_5" ""   
#[4,] "x1" "F_5" ""   
#[5,] ""   ""    "F_7"
#[6,] "x1" ""    "F_7"

Or if there are multiple elements to compare, we could loop over those in lapply, do the rowSums, Reduce to a single logical expression with & and negate (!) for subsetting

all_combs[!Reduce(`&`, lapply(c("F_5", "F_7"),
         function(x) rowSums(all_combs == x) > 0)),]

Another option which is less efficient is to loop over the rows with apply (MARGIN = 1), check if all the values in the vector (c("F_5", "F_7")) are %in% the row, negate (!) to subset

all_combs[!apply(all_combs, 1, function(x) all(c("F_5", "F_7") %in% x)),]

Or use dapply from collapse which is efficient approach for rowwise operations

library(collapse)
sbt(all_combs, !dapply(all_combs, function(x) 
       all(c("F_5", "F_7") %in% x), MARGIN = 1))
#    Var1 Var2  Var3 
#[1,] ""   ""    ""   
#[2,] "x1" ""    ""   
#[3,] ""   "F_5" ""   
#[4,] "x1" "F_5" ""   
#[5,] ""   ""    "F_7"
#[6,] "x1" ""    "F_7"
user_37322710

1) We assume that potential duplicates have an underscore in their name and a common prefix while the other variables do not have underscores in their name. Use sub to remove everything after an underscore in every element of all_combs leaving just the prefix and underscore. Then on each row check if there are any duplicates among the strings having an underscore and reduce all_combs to just those that are not matched.

have_dups <- function(x) anyDuplicated(x[grepl("_", x)]) > 0
ok <- !apply(sub("_.*", "_", all_combs), 1, have_dups)
all_combs[ok, ]

giving:

     Var1 Var2  Var3 
[1,] ""   ""    ""   
[2,] "x1" ""    ""   
[3,] ""   "F_5" ""   
[4,] "x1" "F_5" ""   
[5,] ""   ""    "F_7"
[6,] "x1" ""    "F_7"

2) Another approach would be to just accept the fact that there are duplicates and use the ExhaustiveSearch package to run the regressions. It will automatically run and compare all submodels with combsUpTo or fewer independent variables. This package is very fast and it is likely that the models with duplicates won't come out on top anyways. With t his approach we don't need all_combs or any of the associated code, just this.

library(ExhaustiveSearch)
ExhaustiveSearch(y ~ ., mydata, combsUpTo = 2)
user_5165480
Related