How to solve this error "Type mismatch: cannot convert from List<String> to Iterator<String>"

user_9907185 0
Viewed 3423

I am new in java8 and spark, where I am trying to execute simple flatmap transformation program in java, But I am facing some error in flatmap transformation in second last line Arrays.asList(e.split(" "))); and error is

Type mismatch: cannot convert from List<String> to Iterator<String>

What is the appropriate solution for this problem. Thanks in advance

 import java.util.Arrays;
 import java.util.List;

 import org.apache.spark.SparkConf;
 import org.apache.spark.api.java.JavaRDD;
 import org.apache.spark.api.java.JavaSparkContext;
 import org.apache.commons.lang.StringUtils;

 public class FlatMapExample {
           public static void main(String[] args) throws Exception {
    SparkConf sparkConf = new 
          SparkConf().setMaster("local").setAppName("filter 
          transformation");
    JavaSparkContext sc = new JavaSparkContext(sparkConf);



    // Parallelized with 2 partitions
    JavaRDD<String> rddX = sc.parallelize(
            Arrays.asList("spark rdd example", "sample example"),
            2);

    // map operation will return List of Array in following case
    JavaRDD<String[]> rddY = rddX.map(e -> e.split(" "));

    List<String[]> listUsingMap = rddY.collect();
    for(int i = 0; i < listUsingMap.size(); i++)
    {
        System.out.println("list.."+StringUtils.join(listUsingMap.get(i)));
    }
    //System.out.println("listUsingMap..."+listUsingMap.collect());

    // flatMap operation will return list of String in following case
    JavaRDD<String> rddY2 = rddX.flatMap(e -> Arrays.asList(e.split(" ")));
    List<String> listUsingFlatMap = rddY2.collect();
}

}

java-8 apache-spark java
2 Answers

You should have specified that you are using at least version 2.0 where FlatMapFunction::call returns actually an Iterator and not Iterable (in 1.6 this is the case for example). Thus, your rddX.flatMap is suppose to return an Iterator<String>, while Arrays.asList(e.split(" ")) returns a List<String>.

But there is List::iterator that you can use, as :

 rddX.flatMap(e -> Arrays.asList(e.split(" ")).iterator())
user_10593720

I would like to explain little more, hope would be helpful to you and others who face the same problem.

rdd.flatMap((e -> Arrays.asList(e.split(" "))) is correct in spark 1.6 and in spark 2.0.

rddX.flatMap(e -> Arrays.asList(e.split(" ")).iterator()) is correct.

Below is the flatMap method signature, which takes FlatMapFunction as an parameter.

public <U> JavaRDD<U> flatMap(FlatMapFunction<T, U> f)

Spark 1.6 FlatMapFunction

call method returns Iterable

@FunctionalInterface
public interface FlatMapFunction<T, R> extends Serializable {
   Iterable<R> call(T t) throws Exception
}

Spark 2.0 FlatMapFunction

call method returns Iterator

@FunctionalInterface
public interface FlatMapFunction<T, R> extends Serializable {
    Iterator<R> call(T var1) throws Exception;
}

So, which means flatMap method in spark 1.6 needs a lambda expression (java 8) which returns Iterable while in Spark 2.0 requred return type is Iterator.

Now,

rddX.flatMap(e -> Arrays.asList(e.split(" ")))

Arrays.asList(e.split(" ")) returns a java.util.List and List is an Iterable.

List implements Collection

public interface List<E> extends Collection<E>

Collection implements Iterable.

public interface Collection<E> extends Iterable<E>

So in Spark 1.6 Arrays.asList() works fine but In Spark 2.0 need to further call iterator() on this List to get Iterator.

Arrays.asList(e.split(" ")).iterator()
user_65783050
Related