We are using PySpark 3.1.2.
Every once and a while we are getting the following exception in some random SQL expression:
File "/opt/spark-3.1.2-bin-without-hadoop/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/tmp/spark-0076ad51-6613-4a81-9d2a-795f3ba61fa4/out.zip/mepy_algo/appcode/REM/utilities/spark3_java_logs_utils.py", line 12, in deco
raise converted from e
pyspark.sql.utils.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.internal.SessionStateBuilder':
We try to look what happened on the Java side to print the Java exception and we get the following exception:
Error while instantiating 'org.apache.spark.sql.internal.SessionStateBuilder':
Traceback (most recent call last):
File "/tmp/spark-0076ad51-6613-4a81-9d2a-795f3ba61fa4/out.zip/mepy_algo/appcode/REM/utilities/spark3_java_logs_utils.py", line 8, in deco
return f(*a, **kw)
File "/opt/spark-3.1.2-bin-without-hadoop/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.sql.functions.expr.
: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.internal.SessionStateBuilder':
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1148)
at org.apache.spark.sql.SparkSession.$anonfun$sessionState$2(SparkSession.scala:159)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:155)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:152)
at org.apache.spark.sql.functions$.$anonfun$expr$1(functions.scala:1395)
at scala.Option.map(Option.scala:230)
at org.apache.spark.sql.functions$.expr(functions.scala:1395)
at org.apache.spark.sql.functions.expr(functions.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: LiveListenerBus is stopped.
at org.apache.spark.scheduler.LiveListenerBus.addToQueue(LiveListenerBus.scala:97)
at org.apache.spark.scheduler.LiveListenerBus.addToSharedQueue(LiveListenerBus.scala:70)
at org.apache.spark.sql.util.ExecutionListenerManager.<init>(QueryExecutionListener.scala:80)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.$anonfun$listenerManager$2(BaseSessionStateBuilder.scala:319)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.listenerManager(BaseSessionStateBuilder.scala:319)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:346)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1145)
... 19 more
We need to understand what makes the LiveListenerBus to be stopped because it usually happens after few successful SQL queries/expressions (in minority of the cases it can happen in read.parquet) and there is no other error shown in the logs.
Any idea why this is happening?