I have a scala-spark streaming application consuming kafka TOPICS .
my kafka params looks like this
val kafkaParams = Map[String, Object](
Bootstrap_servers -> "270618767-1-1328258624:9093",
Key_deserializer -> classOf[StringDeserializer],
Value_deserializer -> classOf[StringDeserializer],
Group_id -> config.Group_id,
Auto_offset_reset -> "earliest",
Enable_auto_commit -> (false: java.lang.Boolean),
Security_protocol -> "SSL",
SSL_truststore_location-> "src/main/resources/cert/trust.jks",
SSL_truststore_password -> "****",
SSL_keystore_location -> "src/main/resources/cert/keystore.jks",
SSL_keystore_password -> "*****",
SSL_key_password -> "****",
SSL_KEYSTORE_TYPE_CONFIG -> "JKS",
SSL_TRUSTSTORE_TYPE_CONFIG -> "JKS",
//SSL_KEYSTORE_CERTIFICATE_CHAIN_CONFIG -> scala.io.Source.fromResource("/cert/keystore.jks").mkString,
//SSL_TRUSTSTORE_CERTIFICATES_CONFIG -> scala.io.Source.fromResource("/cert/net.jks").mkString,
)
This setup is working fine in intellij with project structure..
but upon submitting the job into dataproc cluster i am getting java.nio.file.NoSuchFileException:
i have sparkVersion = "2.4.8" , scalaVersion := "2.12.10" and running sbt 1.5.5 with addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")
i am running this assembly into /opt/projectname/lib/assembly.jar folder in dataproc.
