Spark Map task getting stuck due to (?) really big GC times (logs inside), where to go from here?

Viewed 322

I'm doing a project based on http://learningfrombigdata.com/semantic-similarity-between-sentences-using-apache-spark/ with some slight modifications to the code

Here's my code where spark fails (the code fails during the second map, where distance is a fairly computationally heavy function) :

def distance(lvec: String, rvec: String): Double = {
    val l = Vectors.dense(lvec.split(',').map(_.toDouble))
    val r = Vectors.dense(rvec.split(',').map(_.toDouble))
    math.sqrt(Vectors.sqdist(l, r))
}

val wordVectors = wordPairs.map({case (idx, (lword, rword)) => (rword, (idx, lword))})
        .join(w2v) 
        .map({case (rword, ((idx, lword), rvec)) => (lword, (idx, rvec))})
        .join(w2v) 
        .map({case (lword, ((idx, rvec), lvec)) => ((idx, lword), (lvec, rvec))})
        .map({case ((idx, lword), (lvec, rvec)) => ((idx, lword), List(distance(lvec, rvec)))})

Here's where it dies (after thinking for a long while): https://imgur.com/a/yKXIO

I took a look at what the executors were doing and found some weird GC times for the failing ones: https://imgur.com/a/zWhjy

I then enabled verbose GC and writing GC logs for the executors, and here's how they look: https://imgur.com/a/aAjJx

I don't really know much about garbage collection, so I can't quite make sense of this log, but it seems to me that it shouldn't really be doing something called "Full GC" that often.

I would like to know what should I try next:

  1. Tuning GC itself as shown here: https://databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html

  2. Tuning spark parameters (here are my current ones for reference):

  3. Rewriting code

  4. Something else - if I need to post additional info please tell me

0 Answers
Related