I require to use R packages with spark_apply function in the sparklyr package. The rdocumentation is not quite clear. I tried to make spark_apply work by following this link. It worked for the first part with the following modifications.
The working part:
library(sparklyr)
spark_apply_bundle(packages = T, base_path = getwd())
bundle <- paste(getwd(), list.files()[grep("\\.tar$",list.files())][1], sep = "/")
hdfs_path <- "hdfs://<my-ip>/user/hadoop/R/packages/packages.tar"
system("hdfs dfs -moveFromLocal", bundle, "hdfs://<my-ip>/user/hadoop/R/packages")
config <- spark_config()
config$sparklyr.shell.files <- "hdfs://<my-ip>/user/hadoop/R/packages/packages.tar"
sc <- spark_connect(master = "yarn-client",
version = "2.4.0",
config = config)
mtcars_sparklyr <- copy_to(sc, mtcars)
However, when I try to use svm function within the spark_apply, it does not work using the packages argument.
result <- mtcars_sparklyr %>%
spark_apply(
function(d) {
fit <- svm(d$mpg, d$wt)
sum(fit$residuals ^ 2)
},
group_by = "cyl",
packages = bundle
)
The following, on the other hand, works. This is if I pass the svm function within the context. However, I require the packages argument to work because I have several packages and their functions to use within the spark_apply.
result <- mtcars_sparklyr %>%
spark_apply(
function(d) {
fit <- svm(d$mpg, d$wt)
sum(fit$residuals ^ 2)
},
group_by = "cyl",
context = {svm <- e1071::svm}
)