R: Comparing list with grouped values in dataframe; questions about data types

@akrun answered this in the comments, but I love this question, which is really a deep inquiry into R data structures and functional programming, so I'm going to answer it in a longer form here and arrive at the same answer.

First the minimal reproducible problem setup (I simplify version1 and version2 to v1 and v2):

library(tidyverse)

testdf <- data.frame(id = c('a','a','a','b','b'),
                     value = c(1,2,3,1,2))

lookup <- data.frame(col1 = c('x','y','z'),
                     values = c(1,2,3))

values <- lookup$values


v1 <- testdf %>%
  # Make column listing values for each id
  group_by(id) %>%
  summarize(value_list = list(value)) %>%
  ungroup()

v2 <- testdf %>%
  # Nest values into list
  nest(value_list = value)

In v1 and v2, we create a value_list column that is a nested list. The list contents differ (you can tell by the reported dimensions in the OP).

• v1$value_list is a nested atomic vector
• v2$value_list is a nested data.frame

This is because list() -- when passed an atomic vector -- stores that atomic vector as an atomic vector, but nest() when passed an atomic vector, stores this as a data.frame. Why? Because nest() comes from the functional programming paradigm where we use map() (type-safe next gen lapply()) functions to operate on data.frame objects and importantly, return data.frame objects.

Okay, now let's start walking through the solution. It helps to understand how R represents the data types in v1 and v2.

We begin with v1. calling list() on an atomic vector results in a 1D atomic vector. We can pull it out and examine it:

Note that throughout the post I only include code. C/P and run interactively to see output

v1                         # a dataframe
v1$value_list              # a column, which is itself a list (see below)
class(v1$value_list)       # verify this is a list
v1$value_list[[1]]         # b/c it's a list, use [[ notation to pull element 1
class(v1$value_list[[1]])  # it's an atomic vector - "numeric" = integer or double
typeof(v1$value_list[[1]]) # typeof() to show it's double
typeof(1L)                 # as opposed to a strict integer type
length(v1$value_list[[1]]) # atomic vectors have property length

Now we examine v2:

v2                         # a dataframe
v2$value_list              # a column-which is itself a list, see below
class(v2$value_list)       # verify this is a list
v2$value_list[[1]]         # b/c it's a list, use [[ notation to pull element 1
class(v2$value_list[[1]])  # it's a data.frame, NOT an atomic vector!
typeof(v2$value_list[[1]]) # typeof() to show a dataframe is a just special list
dim(v2$value_list[[1]])    # dataframes have property dim
length(v2$value_list[[1]]) # dataframes also have property length, which is ncol()

With that background on object types, let's move to the problem posed in the OP.

And when I try to add this line at the end of each, the output gives all False even though one should be True.

# Mark whether all values exist in value_list for each id 
mutate(values_bool = values %in% value_list)

Essentially, we want to test if all values occur in v1$value_list or v2$value_list. The expected result is TRUE, and then FALSE, for grouping variables a and b respectively. First, we do this the long way, and then we build to a functional programming solution.

# exhaustive approach for list 1 with atomic vectors
values %in% v1$value_list[[1]]       # all values present
all(values %in% v1$value_list[[1]])  # returns TRUE as expected

# exhaustive approach for list 2 with atomic vectors
values %in% v1$value_list[[2]]       # one value missing
all(values %in% v1$value_list[[2]])  # returns FALSE as expected

# now the functional programming solution without indexing. Notes on syntax:
#   - Pass a list (or a vector) as the first argument.
#   - Use ~ to indicate the start of the function to apply across the list
#   - Use .x as a placehold within the function for the ith list element. Think: concise for-loop
map(v1$value_list, ~all(values %in% .x)) # returns c(TRUE, FALSE)

# Two solutions: because we want to output a vector we can unlist(), or 
# use map_lgl() which only returns a boolean and fails if the result is NOT bool
unlist(map(v1$value_list, ~all(values %in% .x))) # returns c(TRUE, FALSE)
map_lgl(v1$value_list, ~all(values %in% .x))     # returns c(TRUE, FALSE)
 
# now the functional programming solution in a pipe (put it all together):
v1 %>% 
  mutate(values_bool = map_lgl(value_list, ~all(values %in% .x)))

# we can do the same with v2, but we need to convert the dataframe to a vector
# so that it works with all() -- this is a bit tedious and verbose, because we 
# used nest() which is designed to work with nested dataframes, but we ultimately 
# want to use all() which takes vectors, so there was no need to nest() in the first
# place. For examples of how to use the power of nest(): 
# https://r4ds.had.co.nz/many-models.html#creating-list-columns
v2 %>% 
  mutate(values_bool = map_lgl(value_list, ~all(values %in% .x$value)))

Additional Resources:

• essential reading on vectors