im trying to run a loop for a 200k dataset. At very beginning, the loop running fast, as time passed, loop performance is getting slower due to increase in dataframe records. After 4 hours of running, only 50k records processed, and the problem is i notice the loop is getting slower and slower.
Question: I was thinking if i can store the output dataframe outside of the loop? so that the loop performance might perform better.
for (Row in 1:nrow(J_table)) {
# Getting rules from Arules package
rules <- apriori(trans,parameter=list(supp=0.001,conf=0.1, target = "rules", minlen=2),appearance=list(lhs=c(as.character(J_table$MF_DESCI[Row])),default="rhs"))
# Convert Rules into dataframe
if(length(rules) != 0){
temp_df <- DATAFRAME(rules)
# J_table Left join temp_df
temp_df<- as.data.frame(cbind(as.character(J_table$SA_TXDATE[Row]),
as.character(J_table$SA_VIPCODE[Row]),
as.character(J_table$SA_DOCNO[Row]),
as.character(J_table$SA_SKU[Row]),
as.character(J_table$MF_DESCI[Row]),
as.character(temp_df$LHS),
as.character(temp_df$RHS),
temp_df$support,
temp_df$confidence,
temp_df$lift,
temp_df$count))
# Append dataframe
temp_df <- rbind(temp_df_1, temp_df)
# Copy to temp_df_1
temp_df_1 <- temp_df
} else {
temp_df_1 <- temp_df_1
}
# distinct temp_Rules_1, make sure unique before next iteration
temp_df_1 <- unique(temp_df_1[1:11])
}
Thanks in advance!