merge large data frame multiple times exhausts memory

Viewed 963

I'm trying to filter a large data frame twice (DF1 and DF2) and then merge the two filtered data frames into one data frame (DF1+DF2->DF3) many times and combine the results into a single data frame (DF=DF3[1]+DF3[2]...DF[n]), but keep running out of memory (8Gb). The initial and final data frames easily fit on a laptop so its the processing that exhausts memory.

What method is fastest and requires least memory? Should I run the code in parts and recombine, get a bigger gun, or is this a job for a relational database or MapReduce?

The code below illustrates the problem.

#create combination df
Combn <- data.frame(t(combn(as.vector(rep(LETTERS[1:26])),2))) %>% 
  mutate_all(as.character)

#create data df
Nrows <- 1000000
Data <- data.frame(Symbol=rep(LETTERS[1:26])) %>% 
  mutate(Symbol=as.character(Symbol)) %>% 
  bind_rows(replicate(Nrows-1,.,simplify=FALSE)) %>% 
  arrange(Symbol) %>% 
  group_by(Symbol) %>% 
  mutate(Idx=seq(1:Nrows)) %>% 
  mutate(Px=round(runif(Nrows)*20)) 

FnPDList <- function(Combn,Data){      
  Dfs <- list()      
  for(i in 1:nrow(Combn)){
    print(i)
    Symbol.1 <- Combn$X1[i]
    Symbol.2 <- Combn$X2[i]

    Sym.2 <- Data %>% 
      filter(Symbol==Symbol.2) 

    Df <- Data %>% 
      filter(Symbol==Symbol.1) %>%
      left_join(Sym.2,by="Idx",suffix=c(".1",".2"))

    Dfs[[i]] <- Df
  }      
  return(Dfs)
}

#splitting into n parts works
X <- FnPDList(slice(Combn,1:10),Data)
Z <- do.call(bind_rows,X) 

#trying to solve in one go exhausts memory
X <- FnPDList(Combn,Data)
Z <- do.call(bind_rows,X) 
0 Answers
Related