I import org.apache.spark.sql.DataFrame in my scala file, than use sbt to compile, the error was object DataFrame is not a member of package org.apache.spark.sql
Searched for some solutions in the Internet, it seems that the problem is spark version is too old. But I am using the newest version(2.1.1), so it is weird.
In REPL, when I import org.apache.spark.sql.DataFrame, there is no error.
My function is like this:
def test(df: DataFrame): Unit={
....
}
When I define this function in REPL, it is fine, but when I compile it using sbt, the error is not found: type DataFrame.
My build.sbt:
name := "Hello"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"
Anyone can help me to fix this problem? Thanks.