java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible

user_3618822 0
Viewed 57

After locally compiling spark v3.2.1, we deployed it in k8s. We observe below stacktrace in spark-master. Any clues to resolve this exception would be helpful.

  • JDK - openjdk8 (1.8)

  • Scala - 2.12.15

  • Hadoop - 3.3.1

    2022-06-23 05:27:25.847 GMT ERROR [SPARK_MASTER] TransportRequestHandler: [rpc-server-4-1] Error while invoking RpcHandler#receive() for one-way message. java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible: stream classdesc serialVersionUID = 6543101073799644159, local class serialVersionUID = 1574364215946805297

spark-streaming apache-spark
1 Answers

Found out that this exception occurs due to scala version incompatibility. Rebuilding after cleaning up the old spark version image in build machine helped to overcome this issue.

user_36188220
Related