I have a Standalone Spark Cluster and try to run TensorFlow On Spark on it using Python. So far, I have just tried very simple examples but I always run into the same problem: Every worker crashes with the same error message:
AttributeError: Can't pickle local object 'start.<locals>.<lambda>'
A new worker is assigned and I am ultimately stuck in an infinite Waiting for reservations... loop. There is no explicit pickling in my program so I guess it must be part of the TensorFlow On Spark pipeline. Without the Spark wrapper my TensorFlow applications run fine. I have tested this both on Windows 7 and CentOS Linux 7 with the same behavior.
The (almost) full error output is the following:
17/08/15 16:40:36 ERROR Executor: Exception in task 0.2 in stage 0.0 (TID 5)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "D:\Spark\python\lib\pyspark.zip\pyspark\worker.py", line 177, in main
File "D:\Spark\python\lib\pyspark.zip\pyspark\worker.py", line 172, in process
File "C:\Program Files\Anaconda3\lib\site-packages\pyspark\rdd.py", line 2423, in pipeline_func
return func(split, prev_func(split, iterator))
File "C:\Program Files\Anaconda3\lib\site-packages\pyspark\rdd.py", line 2423, in pipeline_func
return func(split, prev_func(split, iterator))
File "C:\Program Files\Anaconda3\lib\site-packages\pyspark\rdd.py", line 2423, in pipeline_func
return func(split, prev_func(split, iterator))
File "C:\Program Files\Anaconda3\lib\site-packages\pyspark\rdd.py", line 346, in func
return f(iterator)
File "C:\Program Files\Anaconda3\lib\site-packages\pyspark\rdd.py", line 794, in func
r = f(it)
File "C:\Program Files\Anaconda3\lib\site-packages\tensorflowonspark\TFSparkNode.py", line 290, in _mapfn
TFSparkNode.mgr = TFManager.start(authkey, ['control'], 'remote')
File "C:\Program Files\Anaconda3\lib\site-packages\tensorflowonspark\TFManager.py", line 41, in start
mgr.start()
File "C:\Program Files\Anaconda3\lib\multiprocessing\managers.py", line 513, in start
self._process.start()
File "C:\Program Files\Anaconda3\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Program Files\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
reduction.dump(process_obj, to_child)
File "C:\Program Files\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'start.<locals>.<lambda>'
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
It seems like this issue is known, but no solution exists. Any hint would be appreciated since I am out of ideas.