how to use sparks implicit conversion (e.g. $) in IntelliJ debugger evaluate expression

user_1138523 0
Viewed 252

When debugging Spark/Scala code with IntelliJ, using e.g. df.select($"mycol") does not work in the evaluate expression window, while df.select(col("mycol")) works fine (but needs code change):

enter image description here

It says :

Error during generated code invocation: com.intellij.debugger.engine.evaluation.EvaluateException: Error evaluating method : 'invoke': Method threw 'java.lang.NoSuchFieldError' exception.: Error evaluating method : 'invoke': Method threw 'java.lang.NoSuchFieldError' exception.

Strangely, it seems to work sometimes, especially if the $ is already part of an existing expression in the code which I mark to evaluate. If I write arbitrary expressions (code-fragments), it fails consistently

EDIT: even repeating import spark.implicts._ in the code-fragment window does not help

apache-spark intellij-idea scala
3 Answers

Try this workaround:

import spark.implicits._
$""
df.select($"diff").show()

before

after

user_78035270

It seems that import spark.implicits.StringToColumn at top of the code fragment works

enter image description here

enter image description here

I think the reason is that IntelliJ does not realize that the import of the implicits is used in the first place (it is rendered gray), therefore its not available in the evaluate expression context

user_11385230

I've had a similiar issue with NoSuchFieldError because of missing spark instance within the "Evaluate Expression" dialog. See example below.

class MyClass(implicit spark: SparkSession) {

  import spark.implicits._


  def myFunc1() = {
    // breakpoint here
  }

My workaround was to modify spark instance declaration in the constructor. I've changed "implicit spark" to "implicit val spark". ..and it works :)

user_92621060
Related