How to read a file in worker node in Spark?

user_14002222 0
Viewed 377

I am working on a spark application which needs to read files on the worker nodes. I have created a mapping for my rdd as follows:

        SparkConf conf = new SparkConf().setAppName("startingSpark").setMaster("local[*]");
        JavaSparkContext sc = new JavaSparkContext(conf);              
        JavaRDD<String> file = sc.textFile("D:\\myFile.txt");
        JavaPairRDD<String, String> pairs = file.mapToPair(s -> new Tuple2<>((int)(Integer.parseInt(s.split(",")[1])/100)+"", s.split(",")[1])/100)+"#"+s));

Now, I want to read files based on the key value on each worker node.

JavaPairRDD<String, String> rdd1 = pairs.reduceByKey(new Function2<String, String, String>() {
                
        @Override
        public String call(String v1, String v2) throws Exception {
            String key=v2.split("#")[0];
            JavaRDD <String> br =sc.textFile(key+".txt");
                    return v1+" "+v2;
                }
            });

It gives an error:

java.io.NotSerializableException: org.apache.spark.api.java.JavaSparkContext

I have tried implementing java.io.Serializable with my main class but no luck. Any help is highly appreciated.

spark-streaming apache-spark
1 Answers
java.io.NotSerializableException: org.apache.spark.api.java.JavaSparkContext

The above exception because all your work nodes can't recognize the path "D:\myFile.txt"

You don't need to implement any additional logic on your code, please just use put your source to an HDFS path and then replace to below

JavaRDD<String> file = sc.textFile("hdfs:\\localhost\xx\\xx.txt");
user_64087840
Related