I use spark 1.6.1.
My spark application reads more than 10000 parquet files stored in s3.
val df = sqlContext.read.option("mergeSchema", "true").parquet(myPaths: _*)
myPaths is an Array[String] that contains the paths of the 10000 parquet files. Each path is like this s3n://bucketname/blahblah.parquet
Spark warns message like below.
WARN TaskSetManager: Stage 4 contains a task of very large size (108KB). The maximum recommended task size is 100KB.
Spark has managed to run and finish the job anyway but I guess this can slow down spark processing job.
Does anybody has a good suggestion about this problem?