How do I add a jar using sparklyr?

Question

When I try to access a Hive table using Rstudio and sparklyr using this code:

library(sparklyr)
library(dplyr)
Sys.setenv(SPARK_HOME="/usr/hdp/current/spark2-client") # got from ambari spark2 configs
sc <- spark_connect(master = "yarn-client", config = config, version = '2.1.0')

library(DBI)
tabTweets <- dbGetQuery(sc, "SELECT * FROM tweets0 LIMIT 10")

I get this error:

Error in value[[3L]](cond) : 
  Failed to fetch data: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.openx.data.jsonserde.JsonSerDe

This is because tweets0 was created using JsonSerDe. It is easily fixed, for example, when encountered using Hive CLI by:

Add jar /usr/hdp/2.4.2.0-258/hive/lib/json-serde-1.3.7-jar-with-dependencies.jar;

so how do I do an equivalent add jar using sparklyr?

EDIT: I tried this:

spark_dependencies <- function(spark_version, scala_version, ...) {
  sparklyr::spark_dependency(
    jars = c(
      system.file(
        sprintf("/usr/hdp/2.4.2.0-258/hive/lib/json-serde-1.3.7-jar-with-dependencies.jar"),
    package = "JsonSerDe"
      )
    )
  )
}
.onLoad <- function(libname, pkgname) {
  sparklyr::register_extension(pkgname)
}

library(JsonSerDe)

but still same error and library(JsonSerDe) gives:

Error in library(JsonSerDe) : there is no package called ‘JsonSerDe’

I see nothing in the Spark log about adding a dependency.

How do I add a jar using sparklyr?

2 Answers