Melt basket dataframe to single dataframe without loops

user_7391480 0
Viewed 141

I have a dataframe in basket format like this: 

V1 <- c('milk', 'beer', 'wrench', 'milk' )
V2 <- c('eggs', 'elbow grease', '', 'beer')
V3 <- c('water', '', '', '')

df <- data.frame(V1, V2, V3)

output:

      V1      V2           V3
1   milk      eggs         water
2   beer      elbow grease      
3   wrench                   
4   milk      beer

What I would like to produce is a dataframe in single format like this:

  transaction   product
1           1   milk
2           1   eggs
3           1   water
4           2   beer
5           2   elbow grease
6           3   wrench
7           4   milk
8           4   beer

For now, I want the data in a dataframe so I can filter before I switch to the transactions format that the apriori R package uses.

What is the fastest way to convert this dataframe from basket to single format?

Right now I'm using a loop which is very slow.

dfSingle <- data.frame(product = character(),
                    transaction = integer())
for (row in 1:nrow(df))  {
  # Create a list of products
  productList <- unname(unlist(df[row, ]))

  # Remove blank spaces
  productList <- productList[!productList %in% ""]

  # Convert to a dataframe
  dfTemp <- as.data.frame(productList)
  colnames(dfTemp) <- "product"
  dfTemp$transaction <- row

  # Bind to larger dataframe with previous rows
  dfSingle <- rbind(dfSingle, dfTemp)

}

I've thought about using apply to apply this function to each row, but I'm confused about how to bind the multiple resulting rows to the results of the previous rows.

market-basket-analysis r
5 Answers

You can use stack. The trick is to transpose your data frame, i.e.

df1 <- stack(as.data.frame(t(df), stringsAsFactors = FALSE))

df1[df1$values != '',]
         values ind
#1          milk  V1
#2          eggs  V1
#3         water  V1
#4          beer  V2
#5  elbow grease  V2
#7        wrench  V3
#10         milk  V4
#11         beer  V4

NOTE: A simple rgex can extract only the numbers from ind column, i.e.

df1$ind <- gsub('\\D+', '', df1$ind)

which will give,

         values ind
1          milk   1
2          eggs   1
3         water   1
4          beer   2
5  elbow grease   2
7        wrench   3
10         milk   4
11         beer   4
user_56355800

With tidyverse you can do:

df %>%
 mutate_all(funs(ifelse(. == "", NA_character_, paste0(.)))) %>%
 rowid_to_column(var = "transaction") %>%
 gather(var, product, -transaction, na.rm = TRUE) %>%
 select(-var) %>%
 arrange(transaction)

  transaction      product
1           1         milk
2           1         eggs
3           1        water
4           2         beer
5           2 elbow grease
6           3       wrench
7           4         milk
8           4         beer

First, it is replacing the empty rows with NA_character_. Second, it creates a variable with row ID called "transaction". Third, it transforms the data from wide to long format and also removing the rows with NA_character_. Finally, it arranges the data according "transaction".

user_59645570

or a data.table approach (one-liner)

First melt get the transaction from the rownames: setDT(df)[, transaction := .I ] Then melt, using transaction as id-column: melt( ... , id = "transaction" ) And finally drop the empty values and return the first and third column: ...[!value == "", c(1,3) ]

melt( setDT(df)[, transaction := .I ], id = "transaction" )[!value == "", c(1,3) ]

#    transaction        value
# 1:           1         milk
# 2:           2         beer
# 3:           3       wrench
# 4:           4         milk
# 5:           1         eggs
# 6:           2 elbow grease
# 7:           4         beer
# 8:           1        water
user_63562780

After replacing the character "" with the appropriate format NA, you can create a new column transaction and then used reshape2::melt:

df[df == ""]   <- NA    
df$transaction <- 1:nrow(df)

Then:

melted_df <- na.omit(reshape2::melt(data=df, id.vars="transaction"))

which yields:

> melted_df
  transaction variable        value
1           1       V1         milk
2           2       V1         beer
3           3       V1       wrench
4           4       V1         milk
5           1       V2         eggs
6           2       V2 elbow grease
8           4       V2         beer
9           1       V3        water

The benefit of this function is that it will give you a column variable which gives you the name of the columns of the previous df data.frame. If it's not relevant to you, then remove this column using df$variable <- NULL. If you want to also sort the result by increasing order of transaction: 

out <- melted_df[order(melted_df$transaction), ]

which finally yields:

> out
  transaction        value
1           1         milk
5           1         eggs
9           1        water
2           2         beer
6           2 elbow grease
3           3       wrench
4           4         milk
8           4         beer
user_61653950

Another base R alternative:

do.call(
  rbind, 
  sapply(seq_along(df), function(i) cbind(transaction = i, product = df[[i]][nzchar(df[[i]])])) 
)

     transaction product       
[1,] "1"         "milk"        
[2,] "1"         "beer"        
[3,] "1"         "wrench"      
[4,] "1"         "milk"        
[5,] "2"         "eggs"        
[6,] "2"         "elbow grease"
[7,] "2"         "beer"        
[8,] "3"         "water"
user_45522950
Related