SparkR : How to use a list in summarize

user_15152015 0
Viewed 164

I'm trying to use a list that containing all columns of Spark DataFrame with function last() and put that list in summarize() of grouped DF.

The list is created in this way :

    exprs <- lapply(columns(WORK02), function(x) last(x) %>% alias(x))

And then i trying to make somthing like this :

WORK03 <- WORK02 %>% 
  groupBy(column("AGENZIA")) %>%
  summarize(exprs)

I get the following error :

Error in agg(x, ...) : agg can only support Column or character

If i use per example only one element of that list, it works:

WORK03 <- WORK02 %>% 
  groupBy(column("AGENZIA")) %>%
  summarize(exprs[[1]])

Output :

 AGENZIA SERV
       1    3

But the result that i expect to receive is :

 AGENZIA SERV COLUMN_1 COLUMN_2 COLUMN_3 ..  COLUMN_N
       1    3       V1       V2       V3           VN

Existing an way to do that ?

P.S. Problem is that i can use a limited package of library , only SparkR...

sparkr apache-spark r
1 Answers

The solution I found is the following :

For create list of columns what i am trying to select i used function lapply()

exprs <- lapply(columns(WORK02), function(x) last(x) %>% alias(x))

After that, i removed all null values from the list, (this option is in case if someone want to don't select a specific column, otherwise column will be created as NULL , that will give to us an error after).

exprs <- exprs[!sapply(exprs,is.null)]

And finally i performing groupBy() + summarize() with function do.call() in the following mode:

WORK_TEST1 <- WORK02 %>% 
  groupBy(column("AGENZIA"))

WORK03 <- do.call(summarize,c(WORK_TEST1,exprs))

With this iteration, i receive the result that i'm expecting:

 AGENZIA SERV COLUMN_1 COLUMN_2 COLUMN_3 ..  COLUMN_N
       1    3       V1       V2       V3           VN
user_151520150
Related