convert RDD to Dataset in Java Spark

Question

I have an RDD, i need to convert it into a Dataset, i tried:

Dataset personDS =  sqlContext.createDataset(personRDD, Encoders.bean(Person.class));

the above line throws the error,

cannot resolve method createDataset(org.apache.spark.api.java.JavaRDD Main.Person, org.apache.spark.sql.Encoder T)

however, i can convert to Dataset after converting to Dataframe. the below code works:

Dataset personDF = sqlContext.createDataFrame(personRDD, Person.class);
Dataset personDS = personDF.as(Encoders.bean(Person.class));

user_5462372 · Answer

In addition to accepted answer, if you want to create a Dataset instead of Dataset in Java, please try like this:

StructType yourStruct = ...; //Create your own structtype based on individual field types
Dataset personDS =  sqlContext.createDataset(personRDD.rdd(), RowEncoder.apply(yourStruct));

convert RDD to Dataset in Java Spark

3 Answers