Spark create dataset using encoder where row is array type

user_1970832 0
Viewed 1439

I am unable to figure out, the correct implementation for encoder to create Dataset from below RDD?

e.g.

JavaRDD<Integer[]>rdd= sparkContext.parallelize(
                Arrays.asList(new Integer[][]{new Integer[]{1,2},
                new Integer[]{3,4}
                ,new Integer[]{6,7}}));

Below implementation fails-

DataSet<Integer> ds = sqlContext.createDataset(rdd.rdd(),Encoders.bean(Integer[].class));

Exception in thread "main" java.lang.AssertionError: assertion failed at scala.Predef$.assert(Predef.scala:165) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.javaBean(ExpressionEncoder.scala:90) at org.apache.spark.sql.Encoders$.bean(Encoders.scala:142) at org.apache.spark.sql.Encoders.bean(Encoders.scala)

How to create encode for array type? Thanks!

apache-spark-sql apache-spark java
1 Answers

Please try converting it first to DataFrame and then to Dataset

Dataset<Integer[]> dataFrame = sqlContext.createDataFrame(rdd, Integer[].class);
Dataset<Integer[]> ds = dataFrame.as(Encoders.bean(Integer[].class));
user_120661380
Related