Fastest way to select multiple elements from a list to build a dataframe

user_5749159 0
Viewed 267

I have a list that contains multiple data.frames. I want to select every nth data.frame from the list and combine them into a single data.frame which can be written to a csv.

Here is an example of the list structure: 

one.title <- data.frame(id = '1a', title = 'first title')

one.author <- data.frame(first_name = c('Susan', 'Alice'),
                     last_name  = c('Smith', 'Johnson') )

second.title <- data.frame(id = '2b', title = 'second_title')

second.author <- data.frame(first_name = c('Sarah', 'Mary'),
                        last_name  = c('Davis', 'Proctor') )

one.list <- list()

one.list[[1]]$title <- one.title
one.list[[1]]$author <- one.author
one.list[[2]]$title <- second.title
one.list[[2]]$author <- second.author

Here's my current solution that produces a single data frame for the 'authors' fields:

build_author_table <- function(result.l){

  list_to_df <- function(i){

  x <- result.l[[i]]$author

  return(x)
}


authors_df_l <-(lapply(1:length(result.l), FUN = list_to_df))

authors_df <- do.call("rbind", lapply(authors_df_l, as.data.frame))

return(authors_df)
}

This produces the output I want: 

    first_name last_name
1      Susan     Smith
2      Alice   Johnson
3      Sarah     Davis
4       Mary   Proctor

But as you can probably imagine, when scaled to thousands of records with much larger text fields in the data.frame, it is painfully slow.

Can anyone suggest a faster, more efficient way to produce the final data.frame?

r performance
3 Answers

Your construction code didn't work, but I built one that I think resembles what you're shooting at. 

List of 2
 $ :List of 2
  ..$ title :'data.frame':  1 obs. of  2 variables:
  .. ..$ id   : Factor w/ 1 level "1a": 1
  .. ..$ title: Factor w/ 1 level "first title": 1
  ..$ author:'data.frame':  2 obs. of  2 variables:
  .. ..$ first_name: Factor w/ 2 levels "Alice","Susan": 2 1
  .. ..$ last_name : Factor w/ 2 levels "Johnson","Smith": 2 1
 $ :List of 2
  ..$ title :'data.frame':  1 obs. of  2 variables:
  .. ..$ id   : Factor w/ 1 level "2b": 1
  .. ..$ title: Factor w/ 1 level "second_title": 1
  ..$ author:'data.frame':  2 obs. of  2 variables:
  .. ..$ first_name: Factor w/ 2 levels "Mary","Sarah": 2 1
  .. ..$ last_name : Factor w/ 2 levels "Davis","Proctor": 1 2

If this is what you were thinking of, this works splendidly, you do get a warning because the character strings are factors. These can be ignored, or when building the initial dataframe use stringAsFactors = F as an argument

library(purrr) 
map_dfr(one.list, "author")
user_113367160

Here's a better solution (benchmarked):

data.table::rbindlist(lapply(one.list, "[[", "author"))

The purr solution is pretty, but not that fast. Benchmark results:

microbenchmark(build_author_table(one.list),
    data.table::rbindlist(lapply(one.list, "[[", "author")),
    map_dfr(one.list, "author"))

Unit: microseconds
                                                    expr     min       lq      mean   median       uq        max neval cld
                            build_author_table(one.list) 170.693 190.9460  239.2987 206.4505 272.3815    494.477   100   a
 data.table::rbindlist(lapply(one.list, "[[", "author"))  69.562  88.5590  270.4926  99.1750 152.6735  15068.116   100   a
                             map_dfr(one.list, "author") 214.832 245.2825 2374.5980 281.3210 340.1270 206562.846   100   a
user_84854030

Try this:



one.title <- data.frame(id = '1a', title = 'first title')

one.author <- data.frame(first_name = c('Susan', 'Alice'),
                         last_name  = c('Smith', 'Johnson') )

second.title <- data.frame(id = '2b', title = 'second_title')

second.author <- data.frame(first_name = c('Sarah', 'Mary'),
                            last_name  = c('Davis', 'Proctor') )

one.list <- list(
  list(title = one.title, author =  one.author),
  list(title = second.title, author =  second.author)
)



authors_df_l = lapply(one.list, function(item) item$author)

do.call("rbind",authors_df_l)
user_131190670
Related