Flink cluster not starting due to Could not find or load main class error

user_7362886 0
Viewed 3226

I am trying to set up flink and run a cluster, although I get the following output which seems like the cluster started:

$ ./bin/start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host LAPTOP-HRAHBL24.
Starting taskexecutor daemon on host LAPTOP-HRAHBL24.

When I go to localhost:8081 the connection gets refused, so I checked the flink logs and in the taskexecutor logs I see the following error:

Error: Could not find or load main class org.apache.flink.runtime.taskexecutor.TaskManagerRunner

and in the standalone logs I get this:

Error: Could not find or load main class org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint

I have been searching the internet and couldn't find anything. My java env and system variables are correct as I can see the output of java -version and javac -version. I am using java 8, specifically jdk1.8.0_251. I tried the above with Flink 1.10.1 and 1.5.0, both give me the same error. Any idea on how to resolve this?

apache-flink java
5 Answers

I was experiencing the same issue, but now I am able to start the cluster and see the localhost:8081 UI.

Running a cluster on Windows 10 OS - Apache Flink 1.11.2 for Scala 2.11

here are the steps I took:

  1. Activation WSL for windows : https://www.thewindowsclub.com/how-to-run-sh-or-shell-script-file-in-windows-10 (part 1 Execute Shell Script file using WSL)
  2. Getting Ubuntu for win 10 to run Linux commands (running the .sh files), this can be achieved by going into Microsoft Store and downloading the preferred distribution of Linux. (Or open PowerShell in windows, type bash for easy info)
  3. Installing OpenJDK on the Ubuntu, following: https://askubuntu.com/questions/746413/trying-to-install-java-8-unable-to-locate-package-openjdk-8-jre (first open Linux shell, then follow the instructions)

After these steps you should be able to open Linux Shell in Apache Flink 1.11.2 folder and run ./bin/start-cluster.sh without any problem.

user_61852110

Worked below line updated in flink-daemon.sh

  "$JAVA_RUN" $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath "`manglePathList "$FLINK_TM_CLASSPATH"`" ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 200<&- 2>&1 < /dev/null &
user_105803920

This can be related to the pom.xml file. Here is the sample example:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>flink_examples</groupId>
    <artifactId>kafkaExample</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>Flink Quickstart Job</name>
    <url>http://www.myorganization.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <flink.version>1.9.1</flink.version>
        <java.version>1.8</java.version>
        <scala.binary.version>2.11</scala.binary.version>
        <jackson.version>2.9.0</jackson.version>
        <maven.compiler.source>${java.version}</maven.compiler.source>
        <maven.compiler.target>${java.version}</maven.compiler.target>
    </properties>

    <repositories>
        <repository>
            <id>apache.snapshots</id>
            <name>Apache Development Snapshot Repository</name>
            <url>https://repository.apache.org/content/repositories/snapshots/</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>

    <dependencies>
        <!-- Apache Flink dependencies -->
        <!-- These dependencies are provided, because they should not be packaged into the JAR file. -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>

        <!-- Add connector dependencies here. They must be in the default scope (compile). -->

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>


        <!-- Add logging framework, to produce console output when running in the IDE. -->
        <!-- These dependencies are excluded from the application JAR by default. -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.7.7</version>
<!--            <scope>runtime</scope>-->
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
<!--            <scope>runtime</scope>-->
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>${jackson.version}</version>
        </dependency>
        <dependency>
            <groupId>commons-net</groupId>
            <artifactId>commons-net</artifactId>
            <version>3.7-SNAPSHOT</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>

            <!-- Java Compiler -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>

            <!-- We use the maven-shade plugin to create a fat jar that contains all necessary dependencies. -->
            <!-- Change the value of <mainClass>...</mainClass> if your program entry point changes. -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.0.0</version>
                <executions>
                    <!-- Run shade goal on package phase -->
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <artifactSet>
                                <excludes>
                                    <exclude>org.apache.flink:force-shading</exclude>
                                    <exclude>com.google.code.findbugs:jsr305</exclude>
                                    <exclude>org.slf4j:*</exclude>
                                    <exclude>log4j:*</exclude>
                                </excludes>
                            </artifactSet>
                            <filters>
                                <filter>
                                    <!-- Do not copy the signatures in the META-INF folder.
                                    Otherwise, this might cause SecurityExceptions when using the JAR. -->
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>flink_examples.StreamingJob</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>

        <pluginManagement>
            <plugins>

                <!-- This improves the out-of-the-box experience in Eclipse by resolving some warnings. -->
                <plugin>
                    <groupId>org.eclipse.m2e</groupId>
                    <artifactId>lifecycle-mapping</artifactId>
                    <version>1.0.0</version>
                    <configuration>
                        <lifecycleMappingMetadata>
                            <pluginExecutions>
                                <pluginExecution>
                                    <pluginExecutionFilter>
                                        <groupId>org.apache.maven.plugins</groupId>
                                        <artifactId>maven-shade-plugin</artifactId>
                                        <versionRange>[3.0.0,)</versionRange>
                                        <goals>
                                            <goal>shade</goal>
                                        </goals>
                                    </pluginExecutionFilter>
                                    <action>
                                        <ignore/>
                                    </action>
                                </pluginExecution>
                                <pluginExecution>
                                    <pluginExecutionFilter>
                                        <groupId>org.apache.maven.plugins</groupId>
                                        <artifactId>maven-compiler-plugin</artifactId>
                                        <versionRange>[3.1,)</versionRange>
                                        <goals>
                                            <goal>testCompile</goal>
                                            <goal>compile</goal>
                                        </goals>
                                    </pluginExecutionFilter>
                                    <action>
                                        <ignore/>
                                    </action>
                                </pluginExecution>
                            </pluginExecutions>
                        </lifecycleMappingMetadata>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>

    <!-- This profile helps to make things run out of the box in IntelliJ -->
    <!-- Its adds Flink's core classes to the runtime class path. -->
    <!-- Otherwise they are missing in IntelliJ, because the dependency is 'provided' -->
    <profiles>
        <profile>
            <id>add-dependencies-for-IDEA</id>

            <activation>
                <property>
                    <name>idea.version</name>
                </property>
            </activation>

            <dependencies>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-java</artifactId>
                    <version>${flink.version}</version>
                    <scope>compile</scope>
                </dependency>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
                    <version>${flink.version}</version>
                    <scope>compile</scope>
                </dependency>
            </dependencies>
        </profile>
    </profiles>

</project>
user_89097220

I had the same issue and I've only just solved it.

If you print the java commands the Flink script is trying to execute instead of executing them you'll get something like java <some-flags> -classpath <all-of-the-jars-in-lib-folder>::: <class-to-execute> <more-flags>.

Invoking those commands directly on the shell will get you the same output, but now you can tweak the commands at your will to reproduce bad/desired behaviour. There are two problems with the java commands.

The first problem is the ::: at the end of the classpath, remove it!
The second problem is the log4j.configurationFile flag, it has a path that git bash + windows can't interpret, I had to replace -Dlog4j.configurationFile=file:/c/Users/eduardo/dev/flink-1.11.2/conf/log4j.properties with -Dlog4j.configurationFile="\\Users\\eduardo\\dev\\flink-1.11.2\\conf\\log4j.properties"

That did the trick for me, give it a go and let me know how it went!

user_62209210

I had the same problem, but for me it crashed right after printing Starting standalonesession daemon on host...

I think I also managed to solve this problem, edu's answer here helped me a lot. I'm using this version now: Apache Flink 1.11.2 for Scala 2.11, so it is possible that this workaround is not good for other versions.

So I logged out from where this ::: is coming to the end of the classpath, and it looks like it is from flink-daemon.sh . In this file at the 127th line, there is a command to the JAVA VM, to run some code, it looks like this:

$JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" ${CLASS_TO_RUN} "${ARGS[@]}" > "$out"

The important part of this line is the following:

:$INTERNAL_HADOOP_CLASSPATHS

The 3 colons are coming from here, where the first one is explicitly there, and the second and third are the value of that variable.

So basically just delete this part of the line, and then save the file and start running start-cluster.sh again, this time it should work.

It is possible that it will log some Hadoop related errors, but it won't crash and will work properly. For example I saw this:

Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.

or this:

Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.

But it seems ok, everything is working. Btw installing Hadoop may solve this too, but if it works, I won't modify it. I can open localhost:8081, where the dashboard is runnning.

user_87000020
Related