R for loop, how to store dataframe outside of loop for each iteration to avoid performance drop

Viewed 147

im trying to run a loop for a 200k dataset. At very beginning, the loop running fast, as time passed, loop performance is getting slower due to increase in dataframe records. After 4 hours of running, only 50k records processed, and the problem is i notice the loop is getting slower and slower.

Question: I was thinking if i can store the output dataframe outside of the loop? so that the loop performance might perform better.

for (Row in 1:nrow(J_table)) {

# Getting rules from Arules package
rules <- apriori(trans,parameter=list(supp=0.001,conf=0.1, target = "rules", minlen=2),appearance=list(lhs=c(as.character(J_table$MF_DESCI[Row])),default="rhs"))

# Convert Rules into dataframe
if(length(rules) != 0){
   temp_df <- DATAFRAME(rules)

# J_table Left join temp_df  
   temp_df<- as.data.frame(cbind(as.character(J_table$SA_TXDATE[Row]),
                            as.character(J_table$SA_VIPCODE[Row]),
                            as.character(J_table$SA_DOCNO[Row]),
                            as.character(J_table$SA_SKU[Row]),
                            as.character(J_table$MF_DESCI[Row]),
                            as.character(temp_df$LHS),
                            as.character(temp_df$RHS),
                            temp_df$support, 
                            temp_df$confidence, 
                            temp_df$lift, 
                            temp_df$count)) 

 # Append dataframe
    temp_df <- rbind(temp_df_1, temp_df)

 # Copy to temp_df_1 
    temp_df_1 <- temp_df
} else {
temp_df_1 <- temp_df_1
}

# distinct temp_Rules_1, make sure unique before next iteration
temp_df_1 <- unique(temp_df_1[1:11])
}

Thanks in advance!

1 Answers

I use rbindlist instead of rbind. rbindlist is much faster than rbind.

  1. makes list variable.

    RESULT_DATA <- list()

  2. input the data from each iteration: idx is iteration#.

    RESULT_DATA[[idx]] <- as.data.frame(list([your data]))

  3. outside of the loop, you can bind the result with rbindlist function.

    RETURN_DATA <- rbindlist(RESULT_DATA)

Related