I have my Scala Spark application building fine, but when I add spark-sql dependency to the existing .pom file and build application again, build fails and shows me error in the code which until then was compiling fine.
- Original dependency (when maven build succeeds):
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.4.0-cdh6.3.3</version>
</dependency>
- Dependency I'm adding which causes build to fail:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.4.0-cdh6.3.3</version>
</dependency>
Later, what I did is I removed the new scala-spark code I have written but kept the spark-sql dependency in the pom xml files as is . Should that be a problem? Why existing code build failing just because I added a new dependency.
Below is the new piece of code I want to add:
package com.bell.portrepextracts
import org.apache.spark.sql.{DataFrame, Row, SparkSession}
object SparkAvroToCsv extends App {
val spark: SparkSession = SparkSession.builder
.appName("SparkAvroToCsv").getOrCreate()
val a = spark.read.format("avro")
.load("pathtoavro")//reading avro file
a.write.format("csv")
.option("sep", "\001")
.mode("overwrite")
.option("emptyValue", "\u0000")
.save("targetLoc") // Writing spark DF to hdfs in csv format
}
There are Bunch of error I get in the code which previously was compiling and building fine Like 1)
error: Reference to class Configuration in package conf should not have survived past type checking,
[INFO] it should have been processed and eliminated during expansion of an enclosing macro.
[INFO] val conf = new Configuration()
error: `++` creates a new map. Use `++=` to add an element to this map and return that map itself.
[INFO] val unionMap = agentBusyTimesMap ++ agentChatTimesMap ++ agentConcurrentChatTimesMap