I'm doing a project based on http://learningfrombigdata.com/semantic-similarity-between-sentences-using-apache-spark/ with some slight modifications to the code
Here's my code where spark fails (the code fails during the second map, where distance is a fairly computationally heavy function) :
def distance(lvec: String, rvec: String): Double = {
val l = Vectors.dense(lvec.split(',').map(_.toDouble))
val r = Vectors.dense(rvec.split(',').map(_.toDouble))
math.sqrt(Vectors.sqdist(l, r))
}
val wordVectors = wordPairs.map({case (idx, (lword, rword)) => (rword, (idx, lword))})
.join(w2v)
.map({case (rword, ((idx, lword), rvec)) => (lword, (idx, rvec))})
.join(w2v)
.map({case (lword, ((idx, rvec), lvec)) => ((idx, lword), (lvec, rvec))})
.map({case ((idx, lword), (lvec, rvec)) => ((idx, lword), List(distance(lvec, rvec)))})
Here's where it dies (after thinking for a long while): https://imgur.com/a/yKXIO
I took a look at what the executors were doing and found some weird GC times for the failing ones: https://imgur.com/a/zWhjy
I then enabled verbose GC and writing GC logs for the executors, and here's how they look: https://imgur.com/a/aAjJx
I don't really know much about garbage collection, so I can't quite make sense of this log, but it seems to me that it shouldn't really be doing something called "Full GC" that often.
I would like to know what should I try next:
Tuning GC itself as shown here: https://databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html
Tuning spark parameters (here are my current ones for reference):
Rewriting code
Something else - if I need to post additional info please tell me