How to resolve failed Apache Spark 3.2.1 Maven Test: SparkSubmitCommandBuilderSuite.testPySparkLauncher:154

user_9448569 0
Viewed 611

I am trying to get Apache Spark 3.2.1 up and running, per https://spark.apache.org/docs/latest/building-spark.html

However, I can't pass the test 'Spark Project Launcher' from Maven's ./build/mvn test suite. I am invoking the tests from $SPARK_HOME (/opt/spark). It is failing the 'PySpark' launcher.

Please find details below:

  1. It is for a (highly powered) single server running in standalone mode. OS is Ubuntu 20.04, running on a Docker container. I am SSH into the container when running it.

  2. I have installed the pre-requisites: -Maven 3.6.3 -Java 8 installed via apt-get install openjdk-8-jdk -y -originally did not install Scala 2.12/2.13, as documentation states it's including in the Maven build, however I then did do so (2.12.15) and same issue below.

  • I also pre-installed Hadoop 3.3.1, so that I can use HDFS commands in terminal.
  • Python 3.7.12
  1. The build command I run is : ./build/mvn -Dhadoop.version=${HADOOP_VERSION} -Phive -Phive-thriftserver -Psparkr -DskipTests clean package -e

as I would like to have PySpark and Spark SQL running ('building spark' docs recommand including -phive to ensure this)

  1. My env looks like:

Note: these Yarn variables are populated but it should be running standalone.

JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
HDFS_DATANODE_USER=root
LANG=C.UTF-8
HADOOP_MAPRED_HOME=/opt/hadoop
YARN_HOME=/opt/hadoop
YARN_NODEMANAGER_USER=root
YARN_RESOURCEMANAGER_USER=root
HADOOP_COMMON_HOME=/opt/hadoop
HDFS_SECONDARYNAMENODE_USER=root
PYSPARK_PYTHON=/usr/bin/python3
HADOOP_HDFS_HOME=/opt/hadoop
HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
HADOOP_HOME=/opt/hadoop
PYTHONUTF8=1
SPARK_CONF_DIR=/opt/spark/conf
SPARK_HOME=/opt/spark
JAVA_VERSION=1.8.0_312
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin:/opt/spark/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/hadoop/bin
MAVEN_OPTS=-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g
  1. SPARK CONFIG: spark-env.sh override contains:
export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
export SPARK_EXECUTOR_CORES=1
export SPARK_EXECUTOR_MEMORY=1G
export SPARK_DRIVER_MEMORY=1G
export SPARK_WORKER_MEMORY=2G
export SPARK_DAEMON_MEMORY=1G

Note: I will edit these memory amounts later, once the tests are working first.

  1. HADOOP CONFIG:
core-site.xml properties include:
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>( This is filled in with the server details) </value>
  </property>

  <property>
    <name>hadoop.proxyuser.httpfs.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.httpfs.groups</name>
    <value>*</value>
  </property>

  <property>
    <name>hadoop.proxyuser.hue.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hue.groups</name>
    <value>*</value>
  </property>

hdfs-site.xml properties include:
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.safemode.extension</name>
    <value>0</value>
  </property>
  <property>
     <name>dfs.safemode.min.datanodes</name>
     <value>1</value>
  </property>
  <property>
     <name>dfs.permissions.enabled</name>
     <value>false</value>
  </property>
  <property>
     <name>dfs.safemode.min.datanodes</name>
     <value>1</value>
  </property>
  <property>
     <name>dfs.webhdfs.enabled</name>
     <value>true</value>
  </property>
  <property>
    <name>dfs.name.dir</name>
    <value>( IS SET ) </value>
  </property>
  <property>
    <name>dfs.data.dir</name>
    <value>( IS SET ) </value>
  </property>
  <property>
    <name>dfs.namenode.rpc-bind-host</name>
    <value>( IS SET ) </value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address</name>
    <value>( IS SET ) </value>
  </property>
  <property>
    <name>dfs.namenode.servicerpc-address</name>
    <value>( IS SET ) </value>
  </property>
  <property>
    <name>dfs.namenode.servicerpc-bind-host</name>
    <value> ( IS SET ) </value>
  </property>
</configuration>


<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>(IS SET)</value>
  </property>
  <property>
    <name>mapreduce.framework.name</name>
    <value>local</value>
  </property>
  <property>
    <name>mapred.system.dir</name>
    <value>(IS SET)</value>
  </property>
</configuration>
  1. Dependencies include:
    apt-get install ssh pdsh libdigest-sha-perl openjdk-8-jdk software-properties-common dirmngr libkrb5-dev tini libc6 libpam-modules krb5-user libnss3 procps -y
apt-get install python3-pip -y && \
    pip3 install -U pip && \
    /usr/bin/python3 -m pip install --upgrade pip && \
    pip install --upgrade pandas pyarrow wheel setuptools  cloudpickle requests flake8 flaky pytest krb5 requests-kerberos

  1. I am running as root user. If this is an issue, can you please advise why/ what the implications are for running tests initially, as the documentation does not elaborate.

  2. Spark-submit works if I run a test myself (not the test suite). E.g. the included simple sums of numbers Spark includes.

  3. The error message stack trace for ./build/mvn -X test . Failure point is SparkSubmitCommandBuilderSuite.testPySparkLauncher:154

[INFO]  T E S T S
[INFO] -------------------------------------------------------
[DEBUG] Determined Maven Process ID 1874165
[DEBUG] Fork Channel [1] connection string 'pipe://1' for the implementation class org.apache.maven.plugin.surefire.extensions.LegacyForkChannel
[DEBUG] boot classpath:  /root/.m2/repository/org/apache/maven/surefire/surefire-booter/3.0.0-M5/surefire-booter-3.0.0-M5.jar  /root/.m2/repository/org/apache/maven/surefire/surefire-api/3.0.0-M5/surefire-api-3.0.0-M5.jar  /root/.m2/repository/org/apache/maven/surefire/surefire-logger-api/3.0.0-M5/surefire-logger-api-3.0.0-M5.jar  /root/.m2/repository/org/apache/maven/surefire/surefire-shared-utils/3.0.0-M4/surefire-shared-utils-3.0.0-M4.jar  /root/.m2/repository/org/apache/maven/surefire/surefire-extensions-spi/3.0.0-M5/surefire-extensions-spi-3.0.0-M5.jar  /opt/spark/launcher/target/scala-2.12/test-classes  /opt/spark/launcher/target/scala-2.12/classes  /root/.m2/repository/org/apache/logging/log4j/log4j-api/2.17.1/log4j-api-2.17.1.jar  /root/.m2/repository/org/apache/logging/log4j/log4j-core/2.17.1/log4j-core-2.17.1.jar  /root/.m2/repository/org/apache/logging/log4j/log4j-1.2-api/2.17.1/log4j-1.2-api-2.17.1.jar  /root/.m2/repository/org/mockito/mockito-core/4.2.0/mockito-core-4.2.0.jar  /root/.m2/repository/net/bytebuddy/byte-buddy/1.12.4/byte-buddy-1.12.4.jar  /root/.m2/repository/net/bytebuddy/byte-buddy-agent/1.12.4/byte-buddy-agent-1.12.4.jar  /root/.m2/repository/org/objenesis/objenesis/3.2/objenesis-3.2.jar  /root/.m2/repository/org/slf4j/jul-to-slf4j/1.7.32/jul-to-slf4j-1.7.32.jar  /root/.m2/repository/org/slf4j/slf4j-api/1.7.32/slf4j-api-1.7.32.jar  /root/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar  /root/.m2/repository/org/apache/spark/spark-tags_2.12/3.3.0-SNAPSHOT/spark-tags_2.12-3.3.0-SNAPSHOT.jar  /root/.m2/repository/org/scala-lang/scala-library/2.12.15/scala-library-2.12.15.jar  /root/.m2/repository/org/apache/spark/spark-tags_2.12/3.3.0-SNAPSHOT/spark-tags_2.12-3.3.0-SNAPSHOT-tests.jar  /root/.m2/repository/org/apache/hadoop/hadoop-client-api/3.3.1/hadoop-client-api-3.3.1.jar  /root/.m2/repository/org/apache/hadoop/hadoop-client-runtime/3.3.1/hadoop-client-runtime-3.3.1.jar  /root/.m2/repository/org/apache/htrace/htrace-core4/4.1.0-incubating/htrace-core4-4.1.0-incubating.jar  /root/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar  /root/.m2/repository/com/google/code/findbugs/jsr305/3.0.0/jsr305-3.0.0.jar  /root/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar  /root/.m2/repository/org/scalatest/scalatest_2.12/3.3.0-SNAP3/scalatest_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalatest/scalatest-core_2.12/3.3.0-SNAP3/scalatest-core_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalatest/scalatest-compatible/3.3.0-SNAP3/scalatest-compatible-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalactic/scalactic_2.12/3.3.0-SNAP3/scalactic_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scala-lang/modules/scala-xml_2.12/1.2.0/scala-xml_2.12-1.2.0.jar  /root/.m2/repository/org/scalatest/scalatest-featurespec_2.12/3.3.0-SNAP3/scalatest-featurespec_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalatest/scalatest-flatspec_2.12/3.3.0-SNAP3/scalatest-flatspec_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalatest/scalatest-freespec_2.12/3.3.0-SNAP3/scalatest-freespec_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalatest/scalatest-funsuite_2.12/3.3.0-SNAP3/scalatest-funsuite_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalatest/scalatest-funspec_2.12/3.3.0-SNAP3/scalatest-funspec_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalatest/scalatest-propspec_2.12/3.3.0-SNAP3/scalatest-propspec_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalatest/scalatest-refspec_2.12/3.3.0-SNAP3/scalatest-refspec_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalatest/scalatest-wordspec_2.12/3.3.0-SNAP3/scalatest-wordspec_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalatest/scalatest-diagrams_2.12/3.3.0-SNAP3/scalatest-diagrams_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalatest/scalatest-matchers-core_2.12/3.3.0-SNAP3/scalatest-matchers-core_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalatest/scalatest-shouldmatchers_2.12/3.3.0-SNAP3/scalatest-shouldmatchers_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scalatest/scalatest-mustmatchers_2.12/3.3.0-SNAP3/scalatest-mustmatchers_2.12-3.3.0-SNAP3.jar  /root/.m2/repository/org/scala-lang/scala-reflect/2.12.15/scala-reflect-2.12.15.jar  /root/.m2/repository/org/scalatestplus/scalacheck-1-15_2.12/3.3.0.0-SNAP3/scalacheck-1-15_2.12-3.3.0.0-SNAP3.jar  /root/.m2/repository/org/scalacheck/scalacheck_2.12/1.15.4/scalacheck_2.12-1.15.4.jar  /root/.m2/repository/org/scalatestplus/mockito-4-2_2.12/3.2.11.0/mockito-4-2_2.12-3.2.11.0.jar  /root/.m2/repository/org/scalatestplus/selenium-3-141_2.12/3.3.0.0-SNAP3/selenium-3-141_2.12-3.3.0.0-SNAP3.jar  /root/.m2/repository/org/seleniumhq/selenium/selenium-java/3.141.59/selenium-java-3.141.59.jar  /root/.m2/repository/org/seleniumhq/selenium/selenium-api/3.141.59/selenium-api-3.141.59.jar  /root/.m2/repository/org/seleniumhq/selenium/selenium-chrome-driver/3.141.59/selenium-chrome-driver-3.141.59.jar  /root/.m2/repository/org/seleniumhq/selenium/selenium-edge-driver/3.141.59/selenium-edge-driver-3.141.59.jar  /root/.m2/repository/org/seleniumhq/selenium/selenium-firefox-driver/3.141.59/selenium-firefox-driver-3.141.59.jar  /root/.m2/repository/org/seleniumhq/selenium/selenium-ie-driver/3.141.59/selenium-ie-driver-3.141.59.jar  /root/.m2/repository/org/seleniumhq/selenium/selenium-opera-driver/3.141.59/selenium-opera-driver-3.141.59.jar  /root/.m2/repository/org/seleniumhq/selenium/selenium-remote-driver/3.141.59/selenium-remote-driver-3.141.59.jar  /root/.m2/repository/org/seleniumhq/selenium/selenium-safari-driver/3.141.59/selenium-safari-driver-3.141.59.jar  /root/.m2/repository/org/seleniumhq/selenium/selenium-support/3.141.59/selenium-support-3.141.59.jar  /root/.m2/repository/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar  /root/.m2/repository/com/squareup/okhttp3/okhttp/3.11.0/okhttp-3.11.0.jar  /root/.m2/repository/com/squareup/okio/okio/1.14.0/okio-1.14.0.jar  /root/.m2/repository/org/seleniumhq/selenium/htmlunit-driver/2.50.0/htmlunit-driver-2.50.0.jar  /root/.m2/repository/net/sourceforge/htmlunit/htmlunit/2.50.0/htmlunit-2.50.0.jar  /root/.m2/repository/xalan/xalan/2.7.2/xalan-2.7.2.jar  /root/.m2/repository/xalan/serializer/2.7.2/serializer-2.7.2.jar  /root/.m2/repository/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar  /root/.m2/repository/org/apache/commons/commons-text/1.9/commons-text-1.9.jar  /root/.m2/repository/org/apache/httpcomponents/httpmime/4.5.13/httpmime-4.5.13.jar  /root/.m2/repository/org/apache/httpcomponents/httpclient/4.5.13/httpclient-4.5.13.jar  /root/.m2/repository/org/apache/httpcomponents/httpcore/4.4.14/httpcore-4.4.14.jar  /root/.m2/repository/commons-codec/commons-codec/1.15/commons-codec-1.15.jar  /root/.m2/repository/net/sourceforge/htmlunit/htmlunit-core-js/2.50.0/htmlunit-core-js-2.50.0.jar  /root/.m2/repository/net/sourceforge/htmlunit/neko-htmlunit/2.50.0/neko-htmlunit-2.50.0.jar  /root/.m2/repository/xerces/xercesImpl/2.12.0/xercesImpl-2.12.0.jar  /root/.m2/repository/xml-apis/xml-apis/1.4.01/xml-apis-1.4.01.jar  /root/.m2/repository/net/sourceforge/htmlunit/htmlunit-cssparser/1.7.0/htmlunit-cssparser-1.7.0.jar  /root/.m2/repository/commons-io/commons-io/2.11.0/commons-io-2.11.0.jar  /root/.m2/repository/commons-net/commons-net/3.8.0/commons-net-3.8.0.jar  /root/.m2/repository/org/brotli/dec/0.1.2/dec-0.1.2.jar  /root/.m2/repository/com/shapesecurity/salvation2/3.0.0/salvation2-3.0.0.jar  /root/.m2/repository/org/eclipse/jetty/websocket/websocket-client/9.4.40.v20210413/websocket-client-9.4.40.v20210413.jar  /root/.m2/repository/org/eclipse/jetty/jetty-client/9.4.44.v20210927/jetty-client-9.4.44.v20210927.jar  /root/.m2/repository/org/eclipse/jetty/jetty-http/9.4.44.v20210927/jetty-http-9.4.44.v20210927.jar  /root/.m2/repository/org/eclipse/jetty/jetty-util/9.4.44.v20210927/jetty-util-9.4.44.v20210927.jar  /root/.m2/repository/org/eclipse/jetty/jetty-io/9.4.40.v20210413/jetty-io-9.4.40.v20210413.jar  /root/.m2/repository/org/eclipse/jetty/websocket/websocket-common/9.4.40.v20210413/websocket-common-9.4.40.v20210413.jar  /root/.m2/repository/org/eclipse/jetty/websocket/websocket-api/9.4.40.v20210413/websocket-api-9.4.40.v20210413.jar  /root/.m2/repository/junit/junit/4.13.2/junit-4.13.2.jar  /root/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar  /root/.m2/repository/com/github/sbt/junit-interface/0.13.3/junit-interface-0.13.3.jar  /root/.m2/repository/org/scala-sbt/test-interface/1.0/test-interface-1.0.jar  /root/.m2/repository/org/apache/maven/surefire/surefire-junit4/3.0.0-M5/surefire-junit4-3.0.0-M5.jar  /root/.m2/repository/org/apache/maven/surefire/common-java5/3.0.0-M5/common-java5-3.0.0-M5.jar  /root/.m2/repository/org/apache/maven/surefire/common-junit3/3.0.0-M5/common-junit3-3.0.0-M5.jar  /root/.m2/repository/org/apache/maven/surefire/common-junit4/3.0.0-M5/common-junit4-3.0.0-M5.jar
[DEBUG] boot(compact) classpath:  surefire-booter-3.0.0-M5.jar  surefire-api-3.0.0-M5.jar  surefire-logger-api-3.0.0-M5.jar  surefire-shared-utils-3.0.0-M4.jar  surefire-extensions-spi-3.0.0-M5.jar  test-classes  classes  log4j-api-2.17.1.jar  log4j-core-2.17.1.jar  log4j-1.2-api-2.17.1.jar  mockito-core-4.2.0.jar  byte-buddy-1.12.4.jar  byte-buddy-agent-1.12.4.jar  objenesis-3.2.jar  jul-to-slf4j-1.7.32.jar  slf4j-api-1.7.32.jar  log4j-slf4j-impl-2.17.1.jar  spark-tags_2.12-3.3.0-SNAPSHOT.jar  scala-library-2.12.15.jar  spark-tags_2.12-3.3.0-SNAPSHOT-tests.jar  hadoop-client-api-3.3.1.jar  hadoop-client-runtime-3.3.1.jar  htrace-core4-4.1.0-incubating.jar  commons-logging-1.1.3.jar  jsr305-3.0.0.jar  unused-1.0.0.jar  scalatest_2.12-3.3.0-SNAP3.jar  scalatest-core_2.12-3.3.0-SNAP3.jar  scalatest-compatible-3.3.0-SNAP3.jar  scalactic_2.12-3.3.0-SNAP3.jar  scala-xml_2.12-1.2.0.jar  scalatest-featurespec_2.12-3.3.0-SNAP3.jar  scalatest-flatspec_2.12-3.3.0-SNAP3.jar  scalatest-freespec_2.12-3.3.0-SNAP3.jar  scalatest-funsuite_2.12-3.3.0-SNAP3.jar  scalatest-funspec_2.12-3.3.0-SNAP3.jar  scalatest-propspec_2.12-3.3.0-SNAP3.jar  scalatest-refspec_2.12-3.3.0-SNAP3.jar  scalatest-wordspec_2.12-3.3.0-SNAP3.jar  scalatest-diagrams_2.12-3.3.0-SNAP3.jar  scalatest-matchers-core_2.12-3.3.0-SNAP3.jar  scalatest-shouldmatchers_2.12-3.3.0-SNAP3.jar  scalatest-mustmatchers_2.12-3.3.0-SNAP3.jar  scala-reflect-2.12.15.jar  scalacheck-1-15_2.12-3.3.0.0-SNAP3.jar  scalacheck_2.12-1.15.4.jar  mockito-4-2_2.12-3.2.11.0.jar  selenium-3-141_2.12-3.3.0.0-SNAP3.jar  selenium-java-3.141.59.jar  selenium-api-3.141.59.jar  selenium-chrome-driver-3.141.59.jar  selenium-edge-driver-3.141.59.jar  selenium-firefox-driver-3.141.59.jar  selenium-ie-driver-3.141.59.jar  selenium-opera-driver-3.141.59.jar  selenium-remote-driver-3.141.59.jar  selenium-safari-driver-3.141.59.jar  selenium-support-3.141.59.jar  commons-exec-1.3.jar  okhttp-3.11.0.jar  okio-1.14.0.jar  htmlunit-driver-2.50.0.jar  htmlunit-2.50.0.jar  xalan-2.7.2.jar  serializer-2.7.2.jar  commons-lang3-3.12.0.jar  commons-text-1.9.jar  httpmime-4.5.13.jar  httpclient-4.5.13.jar  httpcore-4.4.14.jar  commons-codec-1.15.jar  htmlunit-core-js-2.50.0.jar  neko-htmlunit-2.50.0.jar  xercesImpl-2.12.0.jar  xml-apis-1.4.01.jar  htmlunit-cssparser-1.7.0.jar  commons-io-2.11.0.jar  commons-net-3.8.0.jar  dec-0.1.2.jar  salvation2-3.0.0.jar  websocket-client-9.4.40.v20210413.jar  jetty-client-9.4.44.v20210927.jar  jetty-http-9.4.44.v20210927.jar  jetty-util-9.4.44.v20210927.jar  jetty-io-9.4.40.v20210413.jar  websocket-common-9.4.40.v20210413.jar  websocket-api-9.4.40.v20210413.jar  junit-4.13.2.jar  hamcrest-core-1.3.jar  junit-interface-0.13.3.jar  test-interface-1.0.jar  surefire-junit4-3.0.0-M5.jar  common-java5-3.0.0-M5.jar  common-junit3-3.0.0-M5.jar  common-junit4-3.0.0-M5.jar
[DEBUG] Forking command line: /bin/sh -c cd /opt/spark/launcher && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=128m -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED -Dio.netty.tryReflectionSetAccessible=true -jar /opt/spark/launcher/target/surefire/surefirebooter724989174307651539.jar /opt/spark/launcher/target/surefire 2022-03-20T00-46-11_265-jvmRun1 surefire4692929599609625015tmp surefire_0469901132567964224tmp
[DEBUG] Fork Channel [1] connected to the client.
[INFO] Running org.apache.spark.launcher.InProcessLauncherSuite
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.604 s - in org.apache.spark.launcher.InProcessLauncherSuite
[INFO] Running org.apache.spark.launcher.SparkSubmitCommandBuilderSuite
[ERROR] Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.045 s <<< FAILURE! - in org.apache.spark.launcher.SparkSubmitCommandBuilderSuite
[ERROR] org.apache.spark.launcher.SparkSubmitCommandBuilderSuite.testPySparkLauncher  Time elapsed: 0 s  <<< FAILURE!
java.lang.AssertionError
        at org.apache.spark.launcher.SparkSubmitCommandBuilderSuite.testPySparkLauncher(SparkSubmitCommandBuilderSuite.java:154)

[INFO] Running org.apache.spark.launcher.LauncherServerSuite
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.043 s - in org.apache.spark.launcher.LauncherServerSuite
[INFO] Running org.apache.spark.launcher.SparkSubmitOptionParserSuite
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.39 s - in org.apache.spark.launcher.SparkSubmitOptionParserSuite
[INFO] Running org.apache.spark.launcher.CommandBuilderUtilsSuite
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.apache.spark.launcher.CommandBuilderUtilsSuite
[INFO] Running org.apache.spark.launcher.ChildProcAppHandleSuite
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.179 s - in org.apache.spark.launcher.ChildProcAppHandleSuite
[DEBUG] Closing the fork 1 after saying GoodBye.
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   SparkSubmitCommandBuilderSuite.testPySparkLauncher:154
[INFO]
[ERROR] Tests run: 46, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Launcher 3.3.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Launcher ............................. FAILURE [  7.156 s]
[INFO] Spark Project Core ................................. SKIPPED
[INFO] Spark Project ML Local Library ..................... SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  8.110 s
[INFO] Finished at: 2022-03-20T00:46:13Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project spark-launcher_2.12: There are test failures.
[ERROR]
[ERROR] Please refer to /opt/spark/launcher/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project spark-launcher_2.12: There are test failures.

Please refer to /opt/spark/launcher/target/surefire-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:972)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures.

Please refer to /opt/spark/launcher/target/surefire-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
    at org.apache.maven.plugin.surefire.SurefireHelper.throwException (SurefireHelper.java:271)
    at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution (SurefireHelper.java:159)
    at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary (SurefirePlugin.java:453)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked (AbstractSurefireMojo.java:1175)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute (AbstractSurefireMojo.java:932)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:972)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Can you please kindly confirm what the issue may be, as I suspect it's something really simple but I can't work this out, and have tried everything I can think of, the official build documentation end to end, and all Stack Overflow resources. The 'sbt' tests seem to run a bit better, but still include a few failures towards the end (it am pretty sure it is able to launch spark though).

I have also reviewed the 'sure fire reports' listed and can't see any issues.

I am not hugely familiar with 'JARS' and Classpaths etc. if there is are any implied setup steps of this nature I may have missed.

Thanks very much for your help, it is much appreciated.

Regards, PB

EDIT: I have since run './build/sbt test', with the following results. Therefore wondering if the above failing is an issue ?

[info] Run completed in 2 hours, 24 minutes, 2 seconds.
[info] Total number of tests run: 3812
[info] Suites: completed 175, aborted 0
[info] Tests: succeeded 3812, failed 0, canceled 2, ignored 19, pending 0
[info] All tests passed.
[error] (core / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] (sql / Test / test) sbt.TestsFailedException: Tests unsuccessful

Noting that at the bottom there are 2 errors listed under 'core' and 'sql' suites (or with the summary are they ok) Thanks?

pyspark apache-spark maven hadoop java
0 Answers
Related