Dependency errors when running Apache Beam with Kafka

user_787036 0
Viewed 299

When I run my Apache Beam code using the Direct Runner I get the following error:

Caused by: java.lang.NoSuchMethodError: org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
    at org.apache.kafka.common.utils.LogContext$LocationAwareKafkaLogger.writeLog (LogContext.java:428)
    at org.apache.kafka.common.utils.LogContext$LocationAwareKafkaLogger.info (LogContext.java:382)
    at org.apache.kafka.clients.consumer.KafkaConsumer.assign (KafkaConsumer.java:1123)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)

Running it as follows works:

mvn package

java -cp target/myjar.jar \
    com.CLASSNAME \
    --runner=DirectRunner \
    ...

Running it as follows does not work:

mvn compile exec:java \
    -Dexec.mainClass="com.CLASSNAME" \
    -Dexec.args="..."

It seems this is related to slf4j dependency conflicts. I've been looking at this for hours and do not seem to make any progress.

Things I've tried:

  • changing versions of libraries
  • putting the slf4j-api dependency on top
  • excluding dependencies
  • changing scopes
  • inspecting dependencies using mvn dependency:tree, but all looks ok
  • changing jdk versions

I have noted the following conflict between identical versions: identical conflict

Any additional ideas or input are appreciated greatly.

Cleaned version of pom file for completeness:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>...</groupId>
    <artifactId>...</artifactId>
    <version>...</version>

    <packaging>jar</packaging>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <beam.version>2.19.0</beam.version>
        <google.cloud.core.version>1.108.1</google.cloud.core.version>
        <maven-compiler-plugin.version>3.7.0</maven-compiler-plugin.version>
        <maven-surefire-plugin.version>2.21.0</maven-surefire-plugin.version>
        <maven-jar-plugin.version>3.0.2</maven-jar-plugin.version>
        <maven-shade-plugin.version>3.1.0</maven-shade-plugin.version>
        <maven-exec-plugin.version>1.6.0</maven-exec-plugin.version>
        <slf4j.version>1.7.30</slf4j.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>
...
    <dependencies>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>${slf4j.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-sdks-java-core</artifactId>
            <version>${beam.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
            <version>${beam.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
            <version>${beam.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
            <version>${beam.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-sdks-java-io-kafka</artifactId>
            <version>${beam.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.4.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>1.9.2</version>
        </dependency>

        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-runners-direct-java</artifactId>
            <version>${beam.version}</version>
        </dependency>
    </dependencies>

</project>
apache-beam google-cloud-dataflow maven
1 Answers

I've managed to resolve the issue. Apparently this was not caused by the dependencies themselves, but rather by a plugin. I omitted the pom.xml in my question as I did not expect this part to have a specific influence.

Cause of error:

<pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <version>${maven-exec-plugin.version}</version>
                    <configuration>
                        <!-- the following is needed for logging to pick up the log4j.properties file -->
                        <includePluginDependencies>true</includePluginDependencies>
                        <cleanupDaemonThreads>false</cleanupDaemonThreads>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>

We had to remove the includePluginDependencies subtag and now it works:

<pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <version>${maven-exec-plugin.version}</version>
                    <configuration>
                        <cleanupDaemonThreads>false</cleanupDaemonThreads>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>

Note that that line was apparently there in order to pick op the log4j property file (see the comment), which we might have to solve otherwise now. We still have to look into whether this gives any issues.

user_7870360
Related