Spark and isolating time taken for tasks

Question

I recently began to use Spark to process huge amount of data (~1TB). And have been able to get the job done too. However I am still trying to understand its working. Consider the following scenario:

Set reference time (say t_ref)
Do any one of the following two tasks:

a. Read large amount of data (~1TB) from tens of thousands of files using SciSpark into RDDs (OR)

b. Read data as above and do additional preprossing work and store the results in a DataFrame
Print the size of the RDD or DataFrame as applicable and time difference wrt to t_ref (ie, t_0a/t_0b)
Do some computation
Save the results

In other words, 1b creates a DataFrame after processing RDDs generated exactly as in 1a.

My query is the following:

Is it correct to infer that t_0b – t_0a = time required for preprocessing? Where can I find an reliable reference for the same?

Edit: Explanation added for the origin of question ...

My suspicion stems from Spark's lazy computation approach and its capability to perform asynchronous jobs. Can/does it initiate subsequent (preprocessing) tasks that can be computed while thousands of input files are being read? The origin of the suspicion is in the unbelievable performance (with results verified okay) I see that look too fantastic to be true.

Thanks for any reply.

user_6933993 · Answer

I believe something like this could assist you (using Scala):

def timeIt[T](op: => T): Float = {
  val start = System.currentTimeMillis
  val res = op
  val end = System.currentTimeMillis
  (end - start) / 1000f
}

def XYZ = {
 val r00 = sc.parallelize(0 to 999999)
 val r01 = r00.map(x => (x,(x,x,x,x,x,x,x)))
 r01.join(r01).count()
}

val time1 = timeIt(XYZ)
// or like this on next line
//val timeN = timeIt(r01.join(r01).count())

println(s"bla bla $time1 seconds.")

You need to be creative and work incrementally with Actions that cause actual execution. This has limitations thus. Lazy evaluation and such.

On the other hand, Spark Web UI records every Action, and records Stage duration for the Action.

In general: performance measuring in shared environments is difficult. Dynamic allocation in Spark in a noisy cluster means that you hold on to acquired resources during the Stage, but upon successive runs of the same or next Stage you may get less resources. But this is at least indicative and you can run in a less busy period.

Spark and isolating time taken for tasks

1 Answers