Does method parameter trigger serialization in Spark?

user_3778765 0
Viewed 264

I read the Spark programming guide about passing functions and wonder what happens when the function reference the outer method parameter/local variable.

For example, I have this object

object Main {
  def main(args: Array[String]): Unit = {
    val ds: Dataset[String] = ???
    ds.map(_ + args(0))
  }
}

Does Spark have to serialize Main? What if args is a local variable inside main?

apache-spark serialization scala
1 Answers

No, in both these cases Spark won't serialize the Main object. Method arguments and local variables (which are pretty much the same thing from the semantics perspective) do not "belong" to the enclosing object or class, they are associated with a particular method invocation and can therefore be captured by the closure directly.

As a general rule, if you need to have a reference to some object in order to access some value, then this reference will be captured and, therefore, serialized:

class Application(n: Int) {
  val x = "internal state " + n

  def doSomething(ds: Dataset[String], param: String): Unit = {
    ds.map(_ + x + param)
  }
}

Note that here in order to access x, which is an instance member, you necessarily have to have the enclosing instance to be available, because it depends on the actual parameters that the instance was constructed with. Another way to see it is to remember that when you use x in the example above, it is actually a shortcut for this.x:

ds.map(_ + this.x + param)

Compared to this, the param value has no such dependency - it is passed as is to the method, and there is no need to access any other enclosing object to use it. Therefore, param will be captured and serialized directly.

That's why there is that advice to put instance members to local variables in order not to capture the entire object: when you put a value to a local variable, it no longer requires accessing the enclosing instance:

val localX = this.x
ds.map(_ + localX + param)

Of course, if you have references to the enclosing instance inside the object that you want to capture, like here:

class Inner(app: Application)

class Application {
  val x = new Inner(this)

  def doSomething(ds: Dataset[String]): Unit = {
    val localX = x
    ds.map(_ + localX.toString)
  }
}

then storing it to a local variable would not help, because Spark would still need to serialize the app field of the Inner class, which points back to the Application instance. That's why you have to be careful if you have complex object graphs that you use in Spark methods which are going to be sent to executors.

user_7882070
Related