How to use packages in spark_apply in R?

Viewed 304

I require to use R packages with spark_apply function in the sparklyr package. The rdocumentation is not quite clear. I tried to make spark_apply work by following this link. It worked for the first part with the following modifications.

The working part:

library(sparklyr)

spark_apply_bundle(packages = T, base_path = getwd())

bundle <- paste(getwd(), list.files()[grep("\\.tar$",list.files())][1], sep = "/")

hdfs_path <- "hdfs://<my-ip>/user/hadoop/R/packages/packages.tar"
system("hdfs dfs -moveFromLocal", bundle, "hdfs://<my-ip>/user/hadoop/R/packages")

config <- spark_config()

config$sparklyr.shell.files <- "hdfs://<my-ip>/user/hadoop/R/packages/packages.tar"

sc <- spark_connect(master = "yarn-client", 
                    version = "2.4.0", 
                    config = config)
mtcars_sparklyr <- copy_to(sc, mtcars)

However, when I try to use svm function within the spark_apply, it does not work using the packages argument.

result <- mtcars_sparklyr %>%
  spark_apply(
    function(d) {
      fit <- svm(d$mpg, d$wt)
      sum(fit$residuals ^ 2)
      },
    group_by = "cyl",
    packages = bundle
  )

The following, on the other hand, works. This is if I pass the svm function within the context. However, I require the packages argument to work because I have several packages and their functions to use within the spark_apply.

result <- mtcars_sparklyr %>%
  spark_apply(
    function(d) {
      fit <- svm(d$mpg, d$wt)
      sum(fit$residuals ^ 2)
    },
    group_by = "cyl",
    context = {svm <- e1071::svm}
  )
0 Answers
Related