Processing several files one-by-one separately by Spark

Viewed 5255

I need help with implementation one workflow with Apache Spark. My task is in next:

  1. I have several CSV files as source data. Note: these files could has different layout
  2. I have metadata with info how I need parse each file (this is not problem)
  3. Main goal: result is source file with several additional columns. I have to update each source file without joining to one output range. For example: source 10 files -> 10 result files and each result file have data only from corresponding source file.

As I know Spark can open many files by mask:

var source = sc.textFile("/source/data*.gz");

But in this case I can't recognize which line of a file. If I get list of source files and try to process by following scenario:

JavaSparkContext sc = new JavaSparkContext(...);
List<String> files = new ArrayList() //list of source files full name's
for(String f : files)
{
   JavaRDD<String> data = sc.textFile(f);
   //process this file with Spark
   outRdd.coalesce(1, true).saveAsTextFile(f + "_out"); 
}

But in this case I will processed all files in sequential mode.

My question is next: how I can processed many files in parallel mode?. For example: one file - one executor?

Thank very much for help me!

2 Answers
Related