Apache Spark core and spark sql

Viewed 149

I have my Scala Spark application building fine, but when I add spark-sql dependency to the existing .pom file and build application again, build fails and shows me error in the code which until then was compiling fine.

  • Original dependency (when maven build succeeds):
<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_2.11</artifactId>
  <version>2.4.0-cdh6.3.3</version>
</dependency>
  • Dependency I'm adding which causes build to fail:
<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-sql_2.11</artifactId>
  <version>2.4.0-cdh6.3.3</version>
</dependency>

Later, what I did is I removed the new scala-spark code I have written but kept the spark-sql dependency in the pom xml files as is . Should that be a problem? Why existing code build failing just because I added a new dependency.

Below is the new piece of code I want to add:

package com.bell.portrepextracts 
import org.apache.spark.sql.{DataFrame, Row, SparkSession} 
   
object SparkAvroToCsv extends App { 

  val spark: SparkSession = SparkSession.builder 
    .appName("SparkAvroToCsv").getOrCreate()
        
  val a = spark.read.format("avro")
    .load("pathtoavro")//reading avro file

  a.write.format("csv")
    .option("sep", "\001")
    .mode("overwrite")
    .option("emptyValue", "\u0000")
    .save("targetLoc") // Writing spark DF to hdfs in csv format

}

There are Bunch of error I get in the code which previously was compiling and building fine Like 1)

error: Reference to class Configuration in package conf should not have survived past type checking,
[INFO] it should have been processed and eliminated during expansion of an enclosing macro.
[INFO]     val conf = new Configuration()
error: `++` creates a new map. Use `++=` to add an element to this map and return that map itself.
[INFO]       val unionMap =  agentBusyTimesMap ++ agentChatTimesMap ++ agentConcurrentChatTimesMap
0 Answers
Related