How to use sortby in Java in Spark

user_1136700 0
Viewed 2541

I have two rdd, and would like to merge together, I have the following question,

I tried the following using union, but union does not sort at all, but I don't know how to use sortby here?

List<Integer> data1 = Arrays.asList(1, 3, 5);
    List<Integer> data2 = Arrays.asList(2, 4, 6, 8);
    JavaRDD<Integer> rdd1 = sc.parallelize(data1);
    JavaRDD<Integer> rdd2 = sc.parallelize(data2);

    JavaRDD<Integer> rdd = rdd1.union(rdd2);
    rdd.sortBy(w->w._1, false);  //compile error

Another question, is there any good way to return the merged the list sorted?

apache-spark
1 Answers

Try below:

List<Integer> data1 = Arrays.asList(1, 3, 5);
        List<Integer> data2 = Arrays.asList(2, 4, 6, 8);
        JavaRDD<Integer> rdd1 = sc.parallelize(data1);
        JavaRDD<Integer> rdd2 = sc.parallelize(data2);

        JavaRDD<Integer> rdd = rdd1.union(rdd2);
        int noofpartitions = 1;
        JavaRDD<Integer> rddSorted = rdd.sortBy(f -> f, true, noofpartitions);
        rddSorted.collect().forEach(f -> System.out.println(f));

sortBy take three parameters: 1. the function 2. a boolean - true represents ascending order, false-descending order, 3. number of partitions It will print :

1
2
3
4
5
6
8

The way you have used is the right one. see this for more details How to merge two presorted rdds in spark?

user_29198490
Related