making the distinction between missing value types (non-response vs skip patterns)

Using NA, Inf, -Inf and NaN we can represent 4 categories of numeric missing values. Below we show the use of NA with Inf and then NA with NaN. In the third approach we discuss the use of naniar package.

1) Recode q2 values of Yes, No, structural missing and missing to 1, 0, Inf and NA respectively. Note that is.na(x) will only report TRUE for an actual NA, is.infinite(x) will only report TRUE for an Inf and !is.finite(x) will report TRUE for NA or Inf in case you need to perform tests. Optionally recode the output back.

df %>% 
  count(q2 = recode(q2, Yes = 1, No = 0, .missing = ifelse(q1 == "No", Inf, NA)))

giving:

# A tibble: 3 x 2
# Groups:   q2 [3]
     q2     n
  <dbl> <int>
1     1     1
2   Inf     2
3    NA     1

2) A variation on this is to use NaN in place of Inf. In that case tests can use is.na(x), is.nan(x) and !is.finite(x)

df %>% 
  count(q2 = recode(q2, Yes = 1, No = 0, .missing = ifelse(q1 == "No", NaN, NA)))

giving:

# A tibble: 3 x 2
     q2     n
  <dbl> <int>
1     1     1
2    NA     1
3   NaN     2

3) The naniar package can create auxilliary columns that define the type of each NA using bind_shadow. We can then recode the auxilliary columns using recode_shadow and then use those in our counting.

library(naniar)

library(naniar)
df %>%
  bind_shadow %>%
  recode_shadow(q2 = .where(is.na(q2) & q1 == "No" ~ "struct")) %>%
  count(q2, q2_NA)

giving:

# A tibble: 3 x 3
  q2    q2_NA         n
  <chr> <fct>     <int>
1 Yes   !NA           1
2 <NA>  NA            1
3 <NA>  NA_struct     2

You could consider to code your variables numerically, which also reduces the file size.

To quickly convert what you already have, do:

df[] <- lapply(df, function(x) ifelse(x == "Yes", 1, 2))
#   q1 q2
# 1  1  1
# 2  1 NA
# 3  2 NA
# 4  2 NA

Different kinds of NA's are usually encoded with unmistakable^* values such as -77 -99.

^{*However, try to replicate some highly cited papers!}

nas <- is.na(df$q2)
df[nas, 'q2'] <- with(df[nas, ], ifelse(q1 == 2, -77, -99))
df
#   q1  q2
# 1  1   1
# 2  1 -99
# 3  2 -77
# 4  2 -77

In your "physical" code book then write:

#   1: "Yes"
#   2: "No"
# -77: "inapplicable"
# -99: "missing"

With this information your data can easily be decoded using factors:

df1 <- df
df1[] <- lapply(df1, function(x) 
  factor(x, levels=c(1, 2, -77, -99), 
         labels=c("Yes", "No", "inapplicable", "missing")))
df1
#    q1           q2
# 1 Yes          Yes
# 2 Yes      missing
# 3  No inapplicable
# 4  No inapplicable

It is obvious that this approach can also be extended to any number of types of missings, e.g.

# -66: not valid
# -88: not asked
# -55: not reached
# ...

Add code book in attributes

It's also possible to add the code book in the attributes. Store information for each variable in a list and Map them on your data frame using `attr<-`.

a <-
  list(q1=c(`1`='Yes', `2`='No', `-77`='inapliccable', `-88`='invalid', `-99`='missing'),
       q1=c(`1`='Yes', `2`='No', `-77`='inapliccable', `-88`='invalid', `-99`='missing'))

df[] <- Map(`attr<-`, df, 'foo', a)

This avoids mistakes when typing in the codebook.

df2 <- df

df2[] <- lapply(df2, function(x) 
  factor(x, levels=names(attr(x, 'foo')), labels=attr(x, 'foo')))
df2
#    q1           q2
# 1 Yes          Yes
# 2 Yes      missing
# 3  No inapliccable
# 4  No inapliccable

R has several special values that behave something like NA: Inf, NaN, etc. But none of them behave exactly like NA, so they are all compromises. A solution that works better (and is much more flexible) was posted here: https://stat.ethz.ch/pipermail/r-devel/2021-May/080776.html .

The idea is to create a new "class" called "MultiMissing" that adds an attribute to distinguish between different kinds of missingness. Here's some code to do it:

asMultiMissing <- function(x) {
  if (isMultiMissing(x))
    return(x)
  missingKind <- ifelse(is.na(x), 1, NA)
  structure(x, 
            missingKind = missingKind, 
            class = c("MultiMissing", class(x)))
}

isMultiMissing <- function(x)
  inherits(x, "MultiMissing")

missingKind <- function(x) {
  if (isMultiMissing(x))
    attr(x, "missingKind")
  else
    ifelse(is.na(x), 1, NA)
}

`missingKind<-` <- function(x, value) {
  class(x) <- setdiff(class(x), "MultiMissing")
  x[!is.na(value)] <- NA
  x <- asMultiMissing(x)
  attr(x, "missingKind") <- value
  x
}

`[.MultiMissing` <- function(x, i, ...) {
  missings <- missingKind(x)
  x <- NextMethod()
  missings <- missings[i]
  missingKind(x) <- missings
  x
}

print.MultiMissing <- function(x, ...) {
  vals <- format(x, ...)
  if (is.character(x) && !inherits(x, "noquote"))
    vals <- ifelse(is.na(x), vals, paste0('"', vals, '"'))
  print(noquote(vals))
  invisible(x)
}

`[<-.MultiMissing` <- function(x, i, value, ...) {
  missings <- missingKind(x)
  class(x) <- setdiff(class(x), "MultiMissing")
  x[i] <- value
  missings[i] <- missingKind(value)
  missingKind(x) <- missings
  x
}

format.MultiMissing <- function(x, ...) {
  missings <- missingKind(x)
  result <- NextMethod()
  ifelse(!is.na(missings),
         paste0("NA.", missings), result)
  
}

pillar_shaft.MultiMissing <- function(x, ...) {
  pillar::pillar_shaft(format(x))
}

q2 <- c("Yes", NA, NA, NA)
missingKind(q2) <- c(NA, "missing", "inapplicable", "inapplicable")
q2
#> [1] "Yes"           NA.missing      NA.inapplicable NA.inapplicable
df <- data.frame(q2 = q2)
df
#>                q2
#> 1             Yes
#> 2      NA.missing
#> 3 NA.inapplicable
#> 4 NA.inapplicable
tib <- tibble::tibble(q2 = q2)
tib
#> # A tibble: 4 × 1
#>   q2             
#>   <MltMssng>     
#> 1 Yes            
#> 2 NA.missing     
#> 3 NA.inapplicable
#> 4 NA.inapplicable

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
df %>% group_by(format(q2)) %>% count()
#> # A tibble: 3 × 2
#> # Groups:   format(q2) [3]
#>   `format(q2)`        n
#>   <chr>           <int>
#> 1 NA.inapplicable     2
#> 2 NA.missing          1
#> 3 Yes                 1

^{Created on 2021-09-14 by the reprex package (v2.0.0)}

You can give a NULL to those where it is not possible to give an answer.

df <- data.frame(q1 = c("Yes", "Yes", "No", "No")
               , q2 = c("Yes", NA, NA, NA))
df$q2[df$q1 != "Yes"] <- list(NULL)

library(dplyr)
df %>% group_by(q2) %>% count() %>% as.data.frame()
#    q2 n
#1  Yes 1
#2   NA 1
#3 NULL 2

Or you give an attr.

df <- data.frame(q1 = c("Yes", "Yes", "No", "No")
               , q2 = c("Yes", NA, NA, NA))
attr(df$q2, "skip") <- ifelse(df$q1 != "Yes", " Skip", "")

df %>% group_by(q2=paste0(q2, attr(q2, "skip"))) %>% count()
#  q2          n
#  <chr>   <int>
#1 NA          1
#2 NA Skip     2
#3 Yes         1