I'm building a Spark (3.0.0) streaming application with Scala 2.12 (built with SBT). How do I get tweets from Twitter API, given all libraries for doing this is for Scala <= 2.11.
EDIT: sample output that I get when trying to built with the libs:
object twitter is not a member of package org.apache.spark.streaming
[error] import org.apache.spark.streaming.twitter.TwitterUtils
In order to import spark-streaming-twitter into your Scala 2.12, you need to specify the Scala version in your import:
resolvers += "twitter_2" at "https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-twitter"
libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.11" % "1.6.3"
This call forces to fetch the scala 2.11 version from artifactory. The import probably tried was:
libraryDependencies += "org.apache.spark" %% "spark-streaming-twitter" % "1.6.3"
The sbt searches for spark-streaming-twitter_2.12 in case your scala build version is 2.12, which does exist in maven atifactory.
Having said that, you need to understand the binary compatibility of scala releases, before you do such thing.
After this import you can call:
import org.apache.spark.streaming.twitter.TwitterUtils