Creating a CDF by Maximizing Prevalence for Increasing Groupings of Dummy Variables (Columns) in R data.table

user_14815134 0
Viewed 28

I have prevalence data by non-exclusive categories/classifications. (e.g., a story could be 'amazing', 'boring', 'charming', 'dark', or any combination of the four.) Illustrative:

library(data.table)

set.seed(0)
results = as.data.table( expand.grid( rep( list(0:1) , 4 ) ) )
names(results) = c('a', 'b', 'c', 'd')
results$prevalence = runif( n = 16 )
results$prevalence = results$prevalence/sum(results$prevalence)

I'd like to be able to answer the question(s):

  1. (trivial) What is the population coverage that is not in any category (a = b = c = d = 0)?
  2. What is the one category that covers the largest percent of the population?
  3. What are the two categories that cover the largest percent of the population?
  4. ... and so on...

Effectively, I'd like to create a quasi-CDF where:

  • I know that for data in the none category (i.e., a = b = c = d = 0) I cover 10% of the population.
  • I know that for data in either one or no categories, I can cover 21% of the population by limiting myself to category c.

That is:

results[ ( a == 0 & b == 0 & d == 0 ) & rowSums( results[ , -'prevalence' ] ) <= 1 , sum(prevalence) ]
  • I know that for data in either two, one, or no categories, I can cover 36% of the population by limiting myself to categories b and c.

That is:

results[ ( a == 0 & d == 0 ) & rowSums( results[ , -'prevalence' ] ) <= 2 , sum(prevalence) ]
  • I know that for data in either three, two, one, or no categories, I can cover 59% of the population by limiting myself to categories a, b, and c.

That is:

results[ ( d == 0 ) & rowSums( results[ , -'prevalence' ] ) <= 3 , sum(prevalence) ]
  • And, trivially, I know that for data in either four, three, two, one, or no categories, I can cover 100% of the population by limiting myself to each of the four categories (a, b, c, d).

In this limited example, I just checked all possible categories to find the largest prevalence by grouping of allowable non-zero categories (actually, as you see by my code snippets, I was doing the inverse and finding prevalence by grouping categories that were restricted to zero).

How can I do this in a data.table way so that I don't have to brute force through the many combinations of dummy variables (columns) in my real summary data set?

I have suspicions that it might involve some clever use of .EACHI or lapply that I haven't been able to think of.

data.table statistics r
1 Answers

Try this:

#' @param dat 'data.frame' (or derivative), with only binary indicator columns
#' @param prev 'numeric', the prevalence indicator to be summed
#' @param n 'integer', number of categories for limiting coverage
#' @return numeric, with attribute "columns" indicating the selected combination of columns
func <- function(dat, prev, n) {
  stopifnot(ncol(dat) >= n)
  if (n == ncol(dat)) {
    out <- sum(prev) # ideally 1
    attr(out, "columns") <- colnames(dat)
  } else {
    com <- t(combn(ncol(dat), ncol(dat) - n))
    vec <- apply(com, 1, function(ind) {
      sum(prev[rowSums(sapply(subset(dat, select = ind), `>`, 0)) < 1])
    })
    out <- max(vec)
    attr(out, "columns") <- colnames(dat)[-com[which.max(vec),]]
  }
  out
}

In action:

func(results[,1:4], results$prevalence, 0)
# [1] 0.1038405
# attr(,"columns")
# character(0)
func(results[,1:4], results$prevalence, 1)
# [1] 0.2090139
# attr(,"columns")
# [1] "c"
func(results[,1:4], results$prevalence, 2)
# [1] 0.3561435
# attr(,"columns")
# [1] "b" "c"
func(results[,1:4], results$prevalence, 3)
# [1] 0.5859805
# attr(,"columns")
# [1] "a" "b" "c"
func(results[,1:4], results$prevalence, 4)
# [1] 1
# attr(,"columns")
# [1] "a" "b" "c" "d"

That is not data.table-syntax, but it is compatible:

results[, func(.SD, prevalence, 2), .SDcols = a:d]
# [1] 0.3561435
# attr(,"columns")
# [1] "b" "c"

or all numbers at once:

results[, sapply(c(0L, seq_along(.SD)), func, dat = .SD, prev = prevalence), .SDcols = a:d]
# [1] 0.1038405 0.2090139 0.3561435 0.5859805 1.0000000

The point of separating the data into separate "category columns" (dat) and a prevalence object is to simplify the column-selection with combn and to not hard-code column names, counts, or positions in the function.

user_33582720
Related