Dictionary is full! error message using dplyr

user_12559770 0
Viewed 1228

Hello I'm truying to do somthing on a dictionary,

here is a head :

           V1 V2 V3  scaf_name
1: scaffold_0  1  1 scaffold_0
2: scaffold_0  2  1 scaffold_0
3: scaffold_0  3  1 scaffold_0
4: scaffold_0  4  1 scaffold_0
5: scaffold_0  5  1 scaffold_0
6: scaffold_0  6  1 scaffold_0

and here is the code I tried:

tab3<-tab %>% 
    group_by(scaf_name) %>%  
    summarise(Avg_group=mean(V3),Length=last(V2))

and here is the error message I got

Error: Internal error: Dictionary is full!

here is the dimension of the tab

> dim(tab)
[1] 852355422         4

So it seems that the dataframe is to huge to use dplyr, does someone know how can I overcome the issue ?

thank you very much

here is a short part of the df

> dput(tab_bis)
structure(list(V1 = c("scaffold_0", "scaffold_0", "scaffold_0", 
"scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", 
"scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", 
"scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", 
"scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", 
"scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", 
"scaffold_0", "scaffold_0"), V2 = 1:30, V3 = c(1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 2L, 2L, 2L, 3L, 4L, 4L, 4L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L), scaf_name = c("scaffold_0", 
"scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", 
"scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", 
"scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", 
"scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", 
"scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0", 
"scaffold_0", "scaffold_0", "scaffold_0", "scaffold_0")), row.names = c(NA, 
-30L), class = c("data.table", "data.frame"), .internal.selfref = <pointer: 0x556f4666b340>)
dplyr dataframe r
2 Answers

This is an issue already known by a tidyverse. https://github.com/r-lib/vctrs/issues/1133 You bypass the limit on a certain value. They have to fix it. ... uint32_t. I thought about just making sure that we store this instead as a uint64_t ... And with example https://github.com/tidyverse/dplyr/issues/5291

My solution will be to use data.table.

library(data.table)
dt = data.table(tab)
dt[,.(Avg_group=mean(V3),Length=last(V2)),by = .(scaf_name)]
user_54425270

Use data tables, it's better for handling huge chunks of data

user_105146820
Related