Pyspark - WARN BisectingKMeans: The input RDD is not directly cached

Question

I'm running bisecting kmeans as

bkm_test=BisectingKMeans().setK(5).setSeed(1)

rdf.cache()
assembled.cache()
model_test=bkm_test.fit(assembled)

I cached the two dataframes as I keep getting the error, but it doesn't make a difference, I found this question which is similar but with kmeans. But I also get a WARN Executor error below. Is this only something inside the algorithm that I can't fix?

17/08/14 21:53:17 WARN BisectingKMeans: The input RDD 306 is not directly cached, which may hurt performance if its parent RDDs are also not cached.
17/08/14 21:53:17 WARN Executor: 1 block locks were not released by TID = 132:
[rdd_302_0]

user_234944 · Answer

This comes from BisectingKMeans within MLlib, which Spark ML uses internally. MLlib uses RDDs of vectors, while Spark ML is DataFrame oriented, so the ML version of BisectingKMeans converts your DataFrame into an RDD of Vector values. The conversion isn't cached, so you end up getting the error.

Hopefully, this isn't a major slowdown. I haven't found an easy way to force the caching of the converted RDD.

Pyspark - WARN BisectingKMeans: The input RDD is not directly cached

1 Answers