I'm trying to filter a large data frame twice (DF1 and DF2) and then merge the two filtered data frames into one data frame (DF1+DF2->DF3) many times and combine the results into a single data frame (DF=DF3[1]+DF3[2]...DF[n]), but keep running out of memory (8Gb). The initial and final data frames easily fit on a laptop so its the processing that exhausts memory.
What method is fastest and requires least memory? Should I run the code in parts and recombine, get a bigger gun, or is this a job for a relational database or MapReduce?
The code below illustrates the problem.
#create combination df
Combn <- data.frame(t(combn(as.vector(rep(LETTERS[1:26])),2))) %>%
mutate_all(as.character)
#create data df
Nrows <- 1000000
Data <- data.frame(Symbol=rep(LETTERS[1:26])) %>%
mutate(Symbol=as.character(Symbol)) %>%
bind_rows(replicate(Nrows-1,.,simplify=FALSE)) %>%
arrange(Symbol) %>%
group_by(Symbol) %>%
mutate(Idx=seq(1:Nrows)) %>%
mutate(Px=round(runif(Nrows)*20))
FnPDList <- function(Combn,Data){
Dfs <- list()
for(i in 1:nrow(Combn)){
print(i)
Symbol.1 <- Combn$X1[i]
Symbol.2 <- Combn$X2[i]
Sym.2 <- Data %>%
filter(Symbol==Symbol.2)
Df <- Data %>%
filter(Symbol==Symbol.1) %>%
left_join(Sym.2,by="Idx",suffix=c(".1",".2"))
Dfs[[i]] <- Df
}
return(Dfs)
}
#splitting into n parts works
X <- FnPDList(slice(Combn,1:10),Data)
Z <- do.call(bind_rows,X)
#trying to solve in one go exhausts memory
X <- FnPDList(Combn,Data)
Z <- do.call(bind_rows,X)