How to find elements with not enough observations in a list

Jesper F 0
Viewed 47

Say I have the following list where each element is a data.frame of different sizes

df1 <- data.frame(matrix(rnorm(12346), ncol = 2))
df2 <- data.frame(matrix(rnorm(14330), ncol = 2))
df3 <- data.frame(matrix(rnorm(2422), ncol = 2))

l <- list(df1, df2, df3)

In my example each data.frame represents a year of observations, and clearly df3 contains a lot fewer observations compared to the other two.

My question is then: What is the best approach to detect those elements of the list l that does not compare in the number of rows and then remove them from the list?

I've so far tried using the median but as this should always remove half of the elements in l I'm not sure this is the best solution for future use

library(collapse)
cutoff <- input %>%
      vapply(nrow, FUN.VALUE = length(.) %>%
      median()
  
idx <- dapply(X = input, FUN = function(x) nrow(x) >= cutoff)
    
input[idx]

where input is a list as the above l

NOTE: As this is my first question on SO, please feel free to edit the question if it does not live up the standards of this community or give feedback on asking better questions. Thanks in advance

EDIT: The question is not so much on how to use median to remove elements of the list, but rather IF median is the right method to remove those data.frames which have a lot less observations than the others

list r
3 Answers

Does this work:

l[sapply(l, function(x) nrow(x) >= median(unlist(lapply(l, nrow))))]
Karthik S0

purrr::keep is the way to go when filtering lists with conditions.

library(purrr)

keep(l, ~ nrow(.x) > median(map_dbl(l, nrow)))
GuedesBF0

It looks like you have a variable number of rows in your data frames, and you want to identify those that are unusually low. This is a statistical problem called outlier detection.

Programmatically, you want to extract the number of rows from your list of data frames, which is easily done with

rows <- sapply(l, nrow)

Statistically, you now want to take a look at your data and its distribution. Good, simple visualisations in R can be

hist(rows)
boxplot(rows)

Note that these will work better if you have many dfs, and are pretty useless with 3.

How to now determine which values are outside an "expected" distribution is not always trivial. Some resources:

Note that it's also acceptable for you to choose a cutoff manually if you can reasonably justify it.

Ottie0
Related