Unseen factor levels when appending new records with unseen string values to a dataframe, cause Warning and result in NA

user_168139 0
Viewed 96901

I have a dataframe (14.5K rows by 15 columns) containing billing data from 2001 to 2007.

I append new 2008 data to it with: alltime <- rbind(alltime,all2008)

Unfortunately that generates a warning:

> Warning message:
In `[<-.factor`(`*tmp*`, ri, value = c(NA, NA, NA, NA, NA, NA, NA,  :
  invalid factor level, NAs generated

My guess is that there are some new patients whose names were not in the previous dataframe and therefore it would not know what level to give those. Similarly new unseen names in the 'referring doctor' column.

What's the solution?

r-factor dataframe append r
7 Answers

When you create the dataframe you have the choice of making your string columns factors (stringsAsFactors=T), or keeping them as strings.

For your case, don't make your string columns factors. Keep them as strings, then appending works fine. If you need them to ultimately be factors, do all the insertion and appending first as string, then finally convert them to factor.

If you make the string columns factors and then append rows containing unseen values, you get the error you mentioned on each new unseen factor level and that value gets replaced with NA...

> df <- data.frame(patient=c('Ann','Bob','Carol'), referring_doctor=c('X','Y','X'), stringsAsFactors=T)

  patient referring_doctor
1     Ann                X
2     Bob                Y
3   Carol                X

> df <- rbind(df, c('Denise','Z'))
Warning messages:
1: In `[<-.factor`(`*tmp*`, ri, value = "Denise") :
  invalid factor level, NA generated
2: In `[<-.factor`(`*tmp*`, ri, value = "Z") :
  invalid factor level, NA generated
> df
  patient referring_doctor
1     Ann                X
2     Bob                Y
3   Carol                X
4    <NA>             <NA>

So don't make your string columns factors. Keep them as strings, then appending works fine:

> df <- data.frame(patient=c('Ann','Bob','Carol'), referring_doctor=c('X','Y','X'), stringsAsFactors=F)
> df <- rbind(df, c('Denise','Z'))
  patient referring_doctor
1     Ann                X
2     Bob                Y
3   Carol                X
4  Denise                Z

To change the default behavior:

options(stringsAsFactors=F)

To convert individual columns to/from string or factor

df$col <- as.character(df$col)
df$col <- as.factor(df$col)
user_2022290
Related