I have a spark RDD defined like this:
val dataset = CreateRDD(data.filter(someFilter))
I observed the following:
//if filter is defined as function, such as following,
//then spark will throw spark `task not serialisable exception`
val someFilter = (some) => true
//if filter is defined as method, such as following then everything will be fine
def someFilter(some) => true
why ?
yes, function/method are all defined as members in the test spec