In Spark using Scala there are different ways to create dataframes, but suppose we have a RDD and for DataFrame's SQL like support I need to convert my RDD to DF , how expensive is this operation ?
Can this operation be optimized ? I Tried converting a RDD[(String,String,String]) containing 5000 items to DF, the conversion took around 600-700ms on my 12 Core CPU, 16 GB RAM and SSD.