I am trying to create a spark session using :
sparkSession = SparkSession.builder().appName(appName).master("local")
.config("hive.metastore.uris", thriftURL).enableHiveSupport().getOrCreate();
But it is giving a NoSuchField Exception as follows:
2020-10-27 20:51:26.963 WARN 11206 --- [ restartedMain] org.apache.hadoop.util.NativeCodeLoader : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-10-27 20:51:27.053 INFO 11206 --- [ restartedMain] org.apache.spark.SparkContext : Submitted application: HdfsPoc
2020-10-27 20:51:27.328 ERROR 11206 --- [ restartedMain] org.apache.spark.SparkContext : Error initializing SparkContext.
java.lang.RuntimeException: java.lang.NoSuchFieldException: DEFAULT_TINY_CACHE_SIZE
at org.apache.spark.network.util.NettyUtils.getPrivateStaticField(NettyUtils.java:131) ~[spark-network-common_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.network.util.NettyUtils.createPooledByteBufAllocator(NettyUtils.java:118) ~[spark-network-common_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.network.server.TransportServer.init(TransportServer.java:95) ~[spark-network-common_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.network.server.TransportServer.<init>(TransportServer.java:74) ~[spark-network-common_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.network.TransportContext.createServer(TransportContext.java:114) ~[spark-network-common_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.rpc.netty.NettyRpcEnv.startServer(NettyRpcEnv.scala:118) ~[spark-core_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.rpc.netty.NettyRpcEnvFactory$$anonfun$4.apply(NettyRpcEnv.scala:457) ~[spark-core_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.rpc.netty.NettyRpcEnvFactory$$anonfun$4.apply(NettyRpcEnv.scala:456) ~[spark-core_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2231) ~[spark-core_2.11-2.2.1.jar:2.2.1]
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) ~[scala-library-2.11.8.jar:na]
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2223) ~[spark-core_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:461) ~[spark-core_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:56) ~[spark-core_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:246) ~[spark-core_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175) ~[spark-core_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257) ~[spark-core_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.SparkContext.<init>(SparkContext.scala:432) ~[spark-core_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516) [spark-core_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918) [spark-sql_2.11-2.2.1.jar:2.2.1]
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910) [spark-sql_2.11-2.2.1.jar:2.2.1]
at scala.Option.getOrElse(Option.scala:121) [scala-library-2.11.8.jar:na]
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910) [spark-sql_2.11-2.2.1.jar:2.2.1]
at com.csg.ipro.util.SparkCommonUtility.createSparkSessionWithHive(SparkCommonUtility.java:54) [classes/:na]
at com.csg.ipro.util.SparkCommonUtility.getSparkSessionWithHive(SparkCommonUtility.java:28) [classes/:na]
at com.csg.ipro.model.FileWatcher.createSession(FileWatcher.java:84) [classes/:na]
at com.csg.ipro.service.WatcherService.startMonitoring(WatcherService.java:43) [classes/:na]
at com.csg.ipro.IproStreamApplication.main(IproStreamApplication.java:21) [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_231]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_231]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_231]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_231]
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) [spring-boot-devtools-2.3.4.RELEASE.jar:2.3.4.RELEASE]
Caused by: java.lang.NoSuchFieldException: DEFAULT_TINY_CACHE_SIZE
at java.lang.Class.getDeclaredField(Class.java:2070) ~[na:1.8.0_231]
at org.apache.spark.network.util.NettyUtils.getPrivateStaticField(NettyUtils.java:127) ~[spark-network-common_2.11-2.2.1.jar:2.2.1]
... 31 common frames omitted
Following are the spark and hadoop dependencies added in the pom.xml file:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.1</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>2.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.10.0</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs-client</artifactId>
<version>2.10.0</version>
<scope>provided</scope>
</dependency>
From whatever I read while trying to find a solution, the dependency versions do not match. But I tried a lot of permutations, none of them worked for me. How can I resolve this error?