Conflict versions when running spark application on Databricks runtime 10.3

Viewed 234

I'm running scala/spark application on Databricks cluster (it tries to read data from an Event Hub topic using EventHub schemaregistry) and getting the following error:

22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-dataformat-xml' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-datatype-jsr310' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/02/17 07:36:14 INFO JacksonVersion: Package versions: jackson-annotations=2.12.3, jackson-core=2.12.3, jackson-databind=2.12.3, jackson-dataformat-xml=1.0.19-SNAPSHOT, jackson-datatype-jsr310=1.0.19-SNAPSHOT, azure-core=1.8.1, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
22/02/17 07:36:14 ERROR ScalaDriverLocal: User Code Stack Trace: 
java.lang.NoSuchMethodError: reactor.netty.http.client.HttpClient.resolver(Lio/netty/resolver/AddressResolverGroup;)Lreactor/netty/transport/ClientTransport;

Looks that maven is trying to use main artificat version (1.0.19-SNAPSHOT) to 'jackson-dataformat-xml' and 'jackson-datatype-jsr310':

22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-dataformat-xml' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/02/17 07:36:14 ERROR JacksonVersion: Version '1.0.19-SNAPSHOT' of package 'jackson-datatype-jsr310' is not supported (older than earliest supported version - `2.10.0`), please upgrade.

The project uses Databricks 10.3 runtime and this pom:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
    <groupId>group</groupId>
    <artifactId>parent</artifactId>
    <version>3.1-SNAPSHOT</version>
</parent>

<artifactId>ingestion</artifactId>
<version>${revision}</version>
<packaging>jar</packaging>
<name>ingestion</name>
<organization>
    <name>company</name>
</organization>

<properties>
    <revision>1.0.19-SNAPSHOT</revision>
    <encoding>UTF-8</encoding>
</properties>

<dependencies>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-core</artifactId>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${scala.compact.version}</artifactId>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.compact.version}</artifactId>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_${scala.compact.version}</artifactId>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-avro_${scala.compact.version}</artifactId>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>io.delta</groupId>
        <artifactId>delta-core_${scala.compact.version}</artifactId>
    </dependency>
    <dependency>
        <groupId>org.scalatest</groupId>
        <artifactId>scalatest_${scala.compact.version}</artifactId>
        <version>3.2.11</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.mockito</groupId>
        <artifactId>mockito-scala_${scala.compact.version}</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.yaml</groupId>
        <artifactId>snakeyaml</artifactId>
    </dependency>
    <dependency>
        <groupId>io.spray</groupId>
        <artifactId>spray-json_${scala.compact.version}</artifactId>
    </dependency>
    <dependency>
        <groupId>com.databricks</groupId>
        <artifactId>dbutils-api_${scala.compact.version}</artifactId>
    </dependency>
    <dependency>
        <groupId>com.github.mrpowers</groupId>
        <artifactId>spark-daria_${scala.compact.version}</artifactId>
    </dependency>
    <dependency>
        <groupId>com.github.mrpowers</groupId>
        <artifactId>spark-fast-tests_${scala.compact.version}</artifactId>
    </dependency>
    <dependency>
        <groupId>joda-time</groupId>
        <artifactId>joda-time</artifactId>
    </dependency>
    <dependency>
        <groupId>io.circe</groupId>
        <artifactId>circe-yaml_${scala.compact.version}</artifactId>
    </dependency>
    <dependency>
        <groupId>io.circe</groupId>
        <artifactId>circe-generic_${scala.compact.version}</artifactId>
    </dependency>
    <dependency>
        <groupId>io.jvm.uuid</groupId>
        <artifactId>scala-uuid_2.12</artifactId>
    </dependency>
    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
    </dependency>
    <dependency>
        <groupId>com.typesafe</groupId>
        <artifactId>config</artifactId>
    </dependency>
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-identity</artifactId>
    </dependency>
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-core</artifactId>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.dataformat</groupId>
        <artifactId>jackson-dataformat-xml</artifactId>
        <version>2.12.3</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.datatype</groupId>
        <artifactId>jackson-datatype-jsr310</artifactId>
        <version>2.12.3</version>
    </dependency>
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-data-schemaregistry</artifactId>
    </dependency>
    <dependency>
        <groupId>commons-cli</groupId>
        <artifactId>commons-cli</artifactId>
    </dependency>
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-core-http-netty</artifactId>
    </dependency>
</dependencies>

<build>
    <resources>
        <resource>
            <directory>src/main/resources</directory>
            <filtering>false</filtering>
        </resource>
        <resource>
            <directory>src/main/resources-filtered</directory>
            <filtering>true</filtering>
        </resource>
    </resources>

    <plugins>
        <plugin>
            <groupId>net.alchim31.maven</groupId>
            <artifactId>scala-maven-plugin</artifactId>
            <executions>
                <execution>
                    <id>scala-compile</id>
                    <goals>
                        <goal>compile</goal>
                        <goal>testCompile</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <scalaVersion>${scala.version}</scalaVersion>
                <jvmArgs>
                    <jvmArg>-Xms64m</jvmArg>
                    <jvmArg>-Xmx1024m</jvmArg>
                </jvmArgs>
            </configuration>
        </plugin>

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <version>3.0.2</version>
            <configuration>
                <archive>
                    <manifestEntries>
                        <Implementation-Title>${project.artifactId}</Implementation-Title>
                        <Implementation-Version>${project.version}</Implementation-Version>
                        <Specification-Vendor>${project.groupId}</Specification-Vendor>
                        <Specification-Title>${project.artifactId}</Specification-Title>
                        <Implementation-Vendor-Id>${project.groupId}</Implementation-Vendor-Id>
                        <Specification-Version>${project.version}</Specification-Version>
                        <Implementation-Vendor>${project.groupId}</Implementation-Vendor>
                    </manifestEntries>
                </archive>
            </configuration>
        </plugin>

        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>flatten-maven-plugin</artifactId>
            <version>1.2.7</version>
            <configuration>
                <updatePomFile>true</updatePomFile>
                <flattenMode>resolveCiFriendliesOnly</flattenMode>
            </configuration>
            <executions>
                <execution>
                    <id>flatten</id>
                    <phase>process-resources</phase>
                    <goals>
                        <goal>flatten</goal>
                    </goals>
                </execution>
                <execution>
                    <id>flatten.clean</id>
                    <phase>clean</phase>
                    <goals>
                        <goal>clean</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>
            <version>3.3.0</version>
            <executions>
                <execution>
                    <id>make-assembly</id>
                    <phase>package</phase>
                    <goals>
                        <goal>single</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <finalName>${project.artifactId}-${project.version}-assembly</finalName>
                <appendAssemblyId>false</appendAssemblyId>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
                <archive>
                    <manifestEntries>
                        <Implementation-Title>${project.artifactId}</Implementation-Title>
                        <Implementation-Version>${project.version}</Implementation-Version>
                        <Specification-Vendor>${project.groupId}</Specification-Vendor>
                        <Specification-Title>${project.artifactId}</Specification-Title>
                        <Implementation-Vendor-Id>${project.groupId}</Implementation-Vendor-Id>
                        <Specification-Version>${project.version}</Specification-Version>
                        <Implementation-Vendor>${project.groupId}</Implementation-Vendor>
                    </manifestEntries>
                </archive>
            </configuration>
        </plugin>

        <plugin>
            <groupId>org.scalatest</groupId>
            <artifactId>scalatest-maven-plugin</artifactId>
            <version>2.0.0</version>
            <configuration>
                <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
            </configuration>
            <executions>
                <execution>
                    <id>test</id>
                    <goals>
                        <goal>test</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

        <plugin>
            <groupId>org.scalastyle</groupId>
            <artifactId>scalastyle-maven-plugin</artifactId>
            <executions>
                <execution>
                    <phase>compile</phase>
                    <goals>
                        <goal>check</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

and this is the parent project:

<groupId>group</groupId>
<artifactId>parent</artifactId>
<version>3.1-SNAPSHOT</version>
<packaging>pom</packaging>
<properties>
    <scala.version>2.12.14</scala.version>
    <scala.compact.version>2.12</scala.compact.version>
    <spark.version>3.2.1</spark.version>
    <databricks.dbutils.api.version>0.0.5</databricks.dbutils.api.version>
    <delta.version>1.1.0</delta.version>
    <postgresql.version>42.2.9</postgresql.version>
    <typesafe.config.version>1.4.0</typesafe.config.version>
    <snakeyaml.version>1.25</snakeyaml.version>
    <spray-json.version>1.3.5</spray-json.version>
    <h2.version>1.4.200</h2.version>
    <mockito.version>1.16.46</mockito.version>
    <scalacheck.version>1.14.0</scalacheck.version>
    <adal4j.version>1.6.6</adal4j.version>
    <scalastyle.version>0.8.0</scalastyle.version>
    <maven-surefire-plugin.version>2.7</maven-surefire-plugin.version>
    <scalatest-maven-plugin.version>1.0</scalatest-maven-plugin.version>
    <spark-daria.version>1.2.3</spark-daria.version>
    <spark-fast-tests.version>1.1.0</spark-fast-tests.version>
    <kafka.version>3.1.0</kafka.version>
    <scala-maven-plugin.version>4.4.0</scala-maven-plugin.version>
    <cats-core.version>2.0.0</cats-core.version>
    <pureconfig_2.11.version>0.12.3</pureconfig_2.11.version>
    <jgitflow-maven-plugin.version>1.0-m5.1</jgitflow-maven-plugin.version>
    <joda-time.version>2.10.6</joda-time.version>
    <mssql-jdbc.version>8.2.2.jre8</mssql-jdbc.version>
    <circe-yaml.artifact>circe-yaml_2.12</circe-yaml.artifact>
    <circe-yaml.version>0.13.1</circe-yaml.version>
    <circe-generic.artifact>circe-generic_2.12</circe-generic.artifact>
    <circe-generic.version>0.13.0</circe-generic.version>
    <scala-uuid_2.12.version>0.3.1</scala-uuid_2.12.version>
    <spark-testing-base.version>0.14.0</spark-testing-base.version>
    <scalatest.version>3.0.5</scalatest.version>
    <httpclient.version>4.5.6</httpclient.version>
    <azure-identity.version>1.4.0</azure-identity.version>
    <azure-core.version>1.23.0</azure-core.version>
    <azure-data-schemaregistry.version>1.0.0</azure-data-schemaregistry.version>
    <log4j-core.version>2.17.1</log4j-core.version>
    <commons-cli.version>1.2</commons-cli.version>
    <reactor-netty.version>1.0.15</reactor-netty.version>
    <azure-core-http-netty.version>1.11.7</azure-core-http-netty.version>
</properties>
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>${log4j-core.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.compact.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.compact.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_${scala.compact.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_${scala.compact.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>${kafka.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-streams</artifactId>
            <version>${kafka.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-databind</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-avro_${scala.compact.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>com.microsoft.azure</groupId>
            <artifactId>adal4j</artifactId>
            <version>${adal4j.version}</version>
        </dependency>
        <dependency>
            <groupId>com.databricks</groupId>
            <artifactId>dbutils-api_2.12</artifactId>
            <version>${databricks.dbutils.api.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-library</artifactId>
                </exclusion>
            </exclusions>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>joda-time</groupId>
            <artifactId>joda-time</artifactId>
            <version>${joda-time.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>${httpclient.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>io.delta</groupId>
            <artifactId>delta-core_${scala.compact.version}</artifactId>
            <version>${delta.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>com.typesafe</groupId>
            <artifactId>config</artifactId>
            <version>${typesafe.config.version}</version>
        </dependency>
        <dependency>
            <groupId>org.yaml</groupId>
            <artifactId>snakeyaml</artifactId>
            <version>${snakeyaml.version}</version>
        </dependency>
        <dependency>
            <groupId>io.circe</groupId>
            <artifactId>${circe-yaml.artifact}</artifactId>
            <version>${circe-yaml.version}</version>
        </dependency>
        <dependency>
            <groupId>io.circe</groupId>
            <artifactId>${circe-generic.artifact}</artifactId>
            <version>${circe-generic.version}</version>
        </dependency>
        <dependency>
            <groupId>io.jvm.uuid</groupId>
            <artifactId>scala-uuid_2.12</artifactId>
            <version>${scala-uuid_2.12.version}</version>
        </dependency>
        <dependency>
            <groupId>io.spray</groupId>
            <artifactId>spray-json_${scala.compact.version}</artifactId>
            <version>${spray-json.version}</version>
        </dependency>
        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-scala_${scala.compact.version}</artifactId>
            <version>${mockito.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.typelevel</groupId>
            <artifactId>cats-core_2.12</artifactId>
            <version>${cats-core.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scalacheck</groupId>
            <artifactId>scalacheck_${scala.compact.version}</artifactId>
            <version>${scalacheck.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.holdenkarau</groupId>
            <artifactId>spark-testing-base_${scala.compact.version}
            <version>2.4.5_${spark-testing-base.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scalatest</groupId>
            <artifactId>scalatest_${scala.compact.version}</artifactId>
            <version>${scalatest.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.github.mrpowers</groupId>
            <artifactId>spark-fast-tests_${scala.compact.version}</artifactId>
            <version>${spark-fast-tests.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.github.mrpowers</groupId>
            <artifactId>spark-daria_${scala.compact.version}</artifactId>
            <version>${spark-daria.version}</version>
        </dependency>
        <dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-identity</artifactId>
            <version>${azure-identity.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.azure</groupId>
                    <artifactId>azure-core-http-netty</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-core</artifactId>
            <version>${azure-core.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.fasterxml.jackson.dataformat</groupId>
                    <artifactId>jackson-dataformat-xml</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.fasterxml.jackson.datatype</groupId>
                    <artifactId>jackson-datatype-jsr310</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.projectreactor.netty</groupId>
                    <artifactId>reactor-netty</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-core-http-netty</artifactId>
            <version>${azure-core-http-netty.version}</version>
        </dependency>
        <dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-data-schemaregistry</artifactId>
            <version>${azure-data-schemaregistry.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.azure</groupId>
                    <artifactId>azure-core-http-netty</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>commons-cli</groupId>
            <artifactId>commons-cli</artifactId>
            <version>${commons-cli.version}</version>
        </dependency>
    </dependencies>
</dependencyManagement>
<build>
    <plugins>
        <plugin>
            <groupId>external.atlassian.jgitflow</groupId>
            <artifactId>jgitflow-maven-plugin</artifactId>
            <version>${jgitflow-maven-plugin.version}</version>
            <configuration>
                <pushFeatures>true</pushFeatures>
                <pushHotfixes>true</pushHotfixes>
                <pushReleases>true</pushReleases>
                <enableSshAgent>true</enableSshAgent>
                <noDeploy>true</noDeploy>
                <flowInitContext>
                    <masterBranchName>master</masterBranchName>
                    <developBranchName>develop</developBranchName>
                    <featureBranchPrefix>feature/</featureBranchPrefix>
                    <releaseBranchPrefix>release/</releaseBranchPrefix>
                    <hotfixBranchPrefix>hotfix/</hotfixBranchPrefix>
                    <versionTagPrefix>${project.artifactId}-</versionTagPrefix>
                </flowInitContext>
            </configuration>
        </plugin>
        <plugin>
            <groupId>net.alchim31.maven</groupId>
            <artifactId>scala-maven-plugin</artifactId>
            <version>${scala-maven-plugin.version}</version>
            <executions>
                <execution>
                    <phase>compile</phase>
                    <goals>
                        <goal>compile</goal>
                        <goal>testCompile</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <jvmArgs>
                    <jvmArg>-Xms1024M</jvmArg>
                    <jvmArg>-Xmx2048M</jvmArg>
                    <jvmArg>-XX:+CMSClassUnloadingEnabled</jvmArg>
                </jvmArgs>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.scalastyle</groupId>
            <artifactId>scalastyle-maven-plugin</artifactId>
            <version>${scalastyle.version}</version>
            <configuration>
                <verbose>true</verbose>
                <failOnViolation>true</failOnViolation>
                <includeTestSourceDirectory>true</includeTestSourceDirectory>
                <failOnWarning>false</failOnWarning>
                <sourceDirectory>${project.basedir}/src/main/scala</sourceDirectory>
                <testSourceDirectory>${project.basedir}/src/test/scala</testSourceDirectory>
                <configLocation>${project.basedir}/scalastyle_config.xml</configLocation>
                <outputFile>${project.basedir}/scalastyle-output.xml</outputFile>
                <outputEncoding>UTF-8</outputEncoding>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>check</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>${maven-surefire-plugin.version}</version>
            <configuration>
                <skipTests>true</skipTests>
            </configuration>
        </plugin>
    </plugins>
</build>
<repositories>
    <repository>
        <id>jitpack.io</id>
        <url>https://jitpack.io</url>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
    <repository>
        <id>dl.binatry.com</id>
        <url>http://dl.bintray.com/spark-packages/maven</url>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
</repositories>

When I run it locally it works fine, the issue arrives when trying to run it in the cluster.

I try to exclude some conflict versions artificats but the issue persists.

Any help?

0 Answers
Related