I have a dataframe consisting of 2,818,615 rows of 388-length pyspark.ml.linalg.SparseVectors and a class label. I'd like to use the pyspark ml RandomForestClassifier using this dataset. Every time I attempt to train a model, spark runs for about 30 minutes before failing because the sparkContext was shut down. If I limit the size of the dataset to just 25K rows, the model is trained successfully, but I need to use the larger dataset.
What could be some troubleshooting steps here?
print(df.rdd.getNumPartitions())
8
df.show()
+--------------------+-----+
| features|label|
+--------------------+-----+
|(388,[1,355,361,3...| 0|
|(388,[1,355,361,3...| 1|
|(388,[1,355,361,3...| 0|
|(388,[1,355,361,3...| 0|
|(388,[1,355,361,3...| 0|
|(388,[1,355,361,3...| 1|
|(388,[1,355,361,3...| 1|
|(388,[1,355,361,3...| 1|
|(388,[1,355,361,3...| 0|
|(388,[1,355,361,3...| 1|
|(388,[1,355,361,3...| 0|
|(388,[1,355,361,3...| 1|
|(388,[1,355,361,3...| 0|
|(388,[1,355,361,3...| 0|
|(388,[1,355,361,3...| 0|
|(388,[1,355,361,3...| 1|
|(388,[1,355,361,3...| 2|
|(388,[1,355,361,3...| 2|
|(388,[1,355,361,3...| 1|
|(388,[1,355,361,3...| 0|
+--------------------+-----+
only showing top 20 rows
My hardware:
- Workers: 4 vCPU, 30.5 GiB memory, 4 instances
- Master: 8 vCPU, 16 GiB memory
Here is how I (try to) train the model:
rf = RandomForestClassifier(featuresCol='features', labelCol='label')
grid = ParamGridBuilder().addGrid(rf.numTrees, [30, 50, 75]).addGrid(rf.maxDepth, [10, 20]).build()
evaluator = MulticlassClassificationEvaluator(metricName="f1")
cv = SparkCV(estimator=rf, estimatorParamMaps=grid, evaluator=evaluator, numFolds=3)
cvModel = cv.fit(df)
The traceback claims the job failed because:
py4j.protocol.Py4JJavaError: An error occurred while calling o417.fit.
: org.apache.spark.SparkException: Job 76 cancelled because SparkContext was shut down
Here are the final few lines of the spark log:
17/11/07 23:15:04 INFO ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 31.
17/11/07 23:15:04 INFO YarnAllocator: Driver requested a total number of 13 executor(s).
17/11/07 23:15:04 INFO ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 14.
17/11/07 23:15:04 INFO YarnAllocator: Driver requested a total number of 12 executor(s).
17/11/07 23:15:04 INFO ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 12.
17/11/07 23:16:21 INFO YarnAllocator: Driver requested a total number of 9 executor(s).
17/11/07 23:16:21 INFO ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 30, 18, 19.
17/11/07 23:20:07 ERROR ApplicationMaster: RECEIVED SIGNAL TERM
17/11/07 23:20:07 INFO ApplicationMaster: Final app status: UNDEFINED, exitCode: 16, (reason: Shutdown hook called before final status was reported.)
17/11/07 23:20:07 INFO ShutdownHookManager: Shutdown hook called