I'm running scala/spark application on Databricks cluster (it tries to read data from an Event Hub topic using EventHub schemaregistry) and getting the following error:
22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-dataformat-xml' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-datatype-jsr310' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/02/17 07:36:14 INFO JacksonVersion: Package versions: jackson-annotations=2.12.3, jackson-core=2.12.3, jackson-databind=2.12.3, jackson-dataformat-xml=1.0.19-SNAPSHOT, jackson-datatype-jsr310=1.0.19-SNAPSHOT, azure-core=1.8.1, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
22/02/17 07:36:14 ERROR ScalaDriverLocal: User Code Stack Trace:
java.lang.NoSuchMethodError: reactor.netty.http.client.HttpClient.resolver(Lio/netty/resolver/AddressResolverGroup;)Lreactor/netty/transport/ClientTransport;
Looks that maven is trying to use main artificat version (1.0.19-SNAPSHOT) to 'jackson-dataformat-xml' and 'jackson-datatype-jsr310':
22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-dataformat-xml' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-datatype-jsr310' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
The project uses Databricks 10.3 runtime and this pom:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>group</groupId>
<artifactId>parent</artifactId>
<version>3.1-SNAPSHOT</version>
</parent>
<artifactId>ingestion</artifactId>
<version>${revision}</version>
<packaging>jar</packaging>
<name>ingestion</name>
<organization>
<name>company</name>
</organization>
<properties>
<revision>1.0.19-SNAPSHOT</revision>
<encoding>UTF-8</encoding>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.compact.version}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.compact.version}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${scala.compact.version}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-avro_${scala.compact.version}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.delta</groupId>
<artifactId>delta-core_${scala.compact.version}</artifactId>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.compact.version}</artifactId>
<version>3.2.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-scala_${scala.compact.version}</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
</dependency>
<dependency>
<groupId>io.spray</groupId>
<artifactId>spray-json_${scala.compact.version}</artifactId>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>dbutils-api_${scala.compact.version}</artifactId>
</dependency>
<dependency>
<groupId>com.github.mrpowers</groupId>
<artifactId>spark-daria_${scala.compact.version}</artifactId>
</dependency>
<dependency>
<groupId>com.github.mrpowers</groupId>
<artifactId>spark-fast-tests_${scala.compact.version}</artifactId>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
</dependency>
<dependency>
<groupId>io.circe</groupId>
<artifactId>circe-yaml_${scala.compact.version}</artifactId>
</dependency>
<dependency>
<groupId>io.circe</groupId>
<artifactId>circe-generic_${scala.compact.version}</artifactId>
</dependency>
<dependency>
<groupId>io.jvm.uuid</groupId>
<artifactId>scala-uuid_2.12</artifactId>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
</dependency>
<dependency>
<groupId>com.typesafe</groupId>
<artifactId>config</artifactId>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-xml</artifactId>
<version>2.12.3</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
<version>2.12.3</version>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-data-schemaregistry</artifactId>
</dependency>
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core-http-netty</artifactId>
</dependency>
</dependencies>
<build>
<resources>
<resource>
<directory>src/main/resources</directory>
<filtering>false</filtering>
</resource>
<resource>
<directory>src/main/resources-filtered</directory>
<filtering>true</filtering>
</resource>
</resources>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<executions>
<execution>
<id>scala-compile</id>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<jvmArgs>
<jvmArg>-Xms64m</jvmArg>
<jvmArg>-Xmx1024m</jvmArg>
</jvmArgs>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
<configuration>
<archive>
<manifestEntries>
<Implementation-Title>${project.artifactId}</Implementation-Title>
<Implementation-Version>${project.version}</Implementation-Version>
<Specification-Vendor>${project.groupId}</Specification-Vendor>
<Specification-Title>${project.artifactId}</Specification-Title>
<Implementation-Vendor-Id>${project.groupId}</Implementation-Vendor-Id>
<Specification-Version>${project.version}</Specification-Version>
<Implementation-Vendor>${project.groupId}</Implementation-Vendor>
</manifestEntries>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>flatten-maven-plugin</artifactId>
<version>1.2.7</version>
<configuration>
<updatePomFile>true</updatePomFile>
<flattenMode>resolveCiFriendliesOnly</flattenMode>
</configuration>
<executions>
<execution>
<id>flatten</id>
<phase>process-resources</phase>
<goals>
<goal>flatten</goal>
</goals>
</execution>
<execution>
<id>flatten.clean</id>
<phase>clean</phase>
<goals>
<goal>clean</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.3.0</version>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<finalName>${project.artifactId}-${project.version}-assembly</finalName>
<appendAssemblyId>false</appendAssemblyId>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifestEntries>
<Implementation-Title>${project.artifactId}</Implementation-Title>
<Implementation-Version>${project.version}</Implementation-Version>
<Specification-Vendor>${project.groupId}</Specification-Vendor>
<Specification-Title>${project.artifactId}</Specification-Title>
<Implementation-Vendor-Id>${project.groupId}</Implementation-Vendor-Id>
<Specification-Version>${project.version}</Specification-Version>
<Implementation-Vendor>${project.groupId}</Implementation-Vendor>
</manifestEntries>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>2.0.0</version>
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
</configuration>
<executions>
<execution>
<id>test</id>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.scalastyle</groupId>
<artifactId>scalastyle-maven-plugin</artifactId>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
and this is the parent project:
<groupId>group</groupId>
<artifactId>parent</artifactId>
<version>3.1-SNAPSHOT</version>
<packaging>pom</packaging>
<properties>
<scala.version>2.12.14</scala.version>
<scala.compact.version>2.12</scala.compact.version>
<spark.version>3.2.1</spark.version>
<databricks.dbutils.api.version>0.0.5</databricks.dbutils.api.version>
<delta.version>1.1.0</delta.version>
<postgresql.version>42.2.9</postgresql.version>
<typesafe.config.version>1.4.0</typesafe.config.version>
<snakeyaml.version>1.25</snakeyaml.version>
<spray-json.version>1.3.5</spray-json.version>
<h2.version>1.4.200</h2.version>
<mockito.version>1.16.46</mockito.version>
<scalacheck.version>1.14.0</scalacheck.version>
<adal4j.version>1.6.6</adal4j.version>
<scalastyle.version>0.8.0</scalastyle.version>
<maven-surefire-plugin.version>2.7</maven-surefire-plugin.version>
<scalatest-maven-plugin.version>1.0</scalatest-maven-plugin.version>
<spark-daria.version>1.2.3</spark-daria.version>
<spark-fast-tests.version>1.1.0</spark-fast-tests.version>
<kafka.version>3.1.0</kafka.version>
<scala-maven-plugin.version>4.4.0</scala-maven-plugin.version>
<cats-core.version>2.0.0</cats-core.version>
<pureconfig_2.11.version>0.12.3</pureconfig_2.11.version>
<jgitflow-maven-plugin.version>1.0-m5.1</jgitflow-maven-plugin.version>
<joda-time.version>2.10.6</joda-time.version>
<mssql-jdbc.version>8.2.2.jre8</mssql-jdbc.version>
<circe-yaml.artifact>circe-yaml_2.12</circe-yaml.artifact>
<circe-yaml.version>0.13.1</circe-yaml.version>
<circe-generic.artifact>circe-generic_2.12</circe-generic.artifact>
<circe-generic.version>0.13.0</circe-generic.version>
<scala-uuid_2.12.version>0.3.1</scala-uuid_2.12.version>
<spark-testing-base.version>0.14.0</spark-testing-base.version>
<scalatest.version>3.0.5</scalatest.version>
<httpclient.version>4.5.6</httpclient.version>
<azure-identity.version>1.4.0</azure-identity.version>
<azure-core.version>1.23.0</azure-core.version>
<azure-data-schemaregistry.version>1.0.0</azure-data-schemaregistry.version>
<log4j-core.version>2.17.1</log4j-core.version>
<commons-cli.version>1.2</commons-cli.version>
<reactor-netty.version>1.0.15</reactor-netty.version>
<azure-core-http-netty.version>1.11.7</azure-core-http-netty.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j-core.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.compact.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.compact.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${scala.compact.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_${scala.compact.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>${kafka.version}</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
<version>${kafka.version}</version>
<exclusions>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-avro_${scala.compact.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>adal4j</artifactId>
<version>${adal4j.version}</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>dbutils-api_2.12</artifactId>
<version>${databricks.dbutils.api.version}</version>
<exclusions>
<exclusion>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</exclusion>
</exclusions>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>${joda-time.version}</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>${httpclient.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.delta</groupId>
<artifactId>delta-core_${scala.compact.version}</artifactId>
<version>${delta.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.typesafe</groupId>
<artifactId>config</artifactId>
<version>${typesafe.config.version}</version>
</dependency>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<version>${snakeyaml.version}</version>
</dependency>
<dependency>
<groupId>io.circe</groupId>
<artifactId>${circe-yaml.artifact}</artifactId>
<version>${circe-yaml.version}</version>
</dependency>
<dependency>
<groupId>io.circe</groupId>
<artifactId>${circe-generic.artifact}</artifactId>
<version>${circe-generic.version}</version>
</dependency>
<dependency>
<groupId>io.jvm.uuid</groupId>
<artifactId>scala-uuid_2.12</artifactId>
<version>${scala-uuid_2.12.version}</version>
</dependency>
<dependency>
<groupId>io.spray</groupId>
<artifactId>spray-json_${scala.compact.version}</artifactId>
<version>${spray-json.version}</version>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-scala_${scala.compact.version}</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.typelevel</groupId>
<artifactId>cats-core_2.12</artifactId>
<version>${cats-core.version}</version>
</dependency>
<dependency>
<groupId>org.scalacheck</groupId>
<artifactId>scalacheck_${scala.compact.version}</artifactId>
<version>${scalacheck.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.holdenkarau</groupId>
<artifactId>spark-testing-base_${scala.compact.version}
<version>2.4.5_${spark-testing-base.version}</version>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.compact.version}</artifactId>
<version>${scalatest.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.github.mrpowers</groupId>
<artifactId>spark-fast-tests_${scala.compact.version}</artifactId>
<version>${spark-fast-tests.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.github.mrpowers</groupId>
<artifactId>spark-daria_${scala.compact.version}</artifactId>
<version>${spark-daria.version}</version>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
<version>${azure-identity.version}</version>
<exclusions>
<exclusion>
<groupId>com.azure</groupId>
<artifactId>azure-core-http-netty</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core</artifactId>
<version>${azure-core.version}</version>
<exclusions>
<exclusion>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-xml</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
</exclusion>
<exclusion>
<groupId>io.projectreactor.netty</groupId>
<artifactId>reactor-netty</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core-http-netty</artifactId>
<version>${azure-core-http-netty.version}</version>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-data-schemaregistry</artifactId>
<version>${azure-data-schemaregistry.version}</version>
<exclusions>
<exclusion>
<groupId>com.azure</groupId>
<artifactId>azure-core-http-netty</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
<version>${commons-cli.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>external.atlassian.jgitflow</groupId>
<artifactId>jgitflow-maven-plugin</artifactId>
<version>${jgitflow-maven-plugin.version}</version>
<configuration>
<pushFeatures>true</pushFeatures>
<pushHotfixes>true</pushHotfixes>
<pushReleases>true</pushReleases>
<enableSshAgent>true</enableSshAgent>
<noDeploy>true</noDeploy>
<flowInitContext>
<masterBranchName>master</masterBranchName>
<developBranchName>develop</developBranchName>
<featureBranchPrefix>feature/</featureBranchPrefix>
<releaseBranchPrefix>release/</releaseBranchPrefix>
<hotfixBranchPrefix>hotfix/</hotfixBranchPrefix>
<versionTagPrefix>${project.artifactId}-</versionTagPrefix>
</flowInitContext>
</configuration>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>${scala-maven-plugin.version}</version>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<jvmArgs>
<jvmArg>-Xms1024M</jvmArg>
<jvmArg>-Xmx2048M</jvmArg>
<jvmArg>-XX:+CMSClassUnloadingEnabled</jvmArg>
</jvmArgs>
</configuration>
</plugin>
<plugin>
<groupId>org.scalastyle</groupId>
<artifactId>scalastyle-maven-plugin</artifactId>
<version>${scalastyle.version}</version>
<configuration>
<verbose>true</verbose>
<failOnViolation>true</failOnViolation>
<includeTestSourceDirectory>true</includeTestSourceDirectory>
<failOnWarning>false</failOnWarning>
<sourceDirectory>${project.basedir}/src/main/scala</sourceDirectory>
<testSourceDirectory>${project.basedir}/src/test/scala</testSourceDirectory>
<configLocation>${project.basedir}/scalastyle_config.xml</configLocation>
<outputFile>${project.basedir}/scalastyle-output.xml</outputFile>
<outputEncoding>UTF-8</outputEncoding>
</configuration>
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>jitpack.io</id>
<url>https://jitpack.io</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<id>dl.binatry.com</id>
<url>http://dl.bintray.com/spark-packages/maven</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
When I run it locally it works fine, the issue arrives when trying to run it in the cluster.
I try to exclude some conflict versions artificats but the issue persists.
Any help?