Replicating a row from a Dataset n times in Apache Spark using Java

user_4431568 0
Viewed 892

I'm trying to replicate a single row from a Dataset n times and create a new Dataset from it. But, while replicating I need a column's value to be changed for each replication since it would be end up as the primary key when stored finally.

Below is the Scala code from SO post : Replicate Spark Row N-times

import org.apache.spark.sql.functions._

val result = singleRowDF
  .withColumn("dummy", explode(array((1 until 100).map(lit): _*)))
  .selectExpr(singleRowDF.columns: _*)

How can I create a column from an array of values in Java and pass it to explode function? Suggestions are helpful.

Thanks

spark-dataframe apache-spark-dataset apache-spark java
1 Answers

This is the Java program to replicate a single row from a Dataset n times.

import static org.apache.spark.sql.functions.col;
import static org.apache.spark.sql.functions.explode;
import static org.apache.spark.sql.functions.lit;

import java.util.ArrayList;
import java.util.List;
import java.util.stream.IntStream;

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Encoders;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;

public class SparkSample{
    public static void main(String[] args) {
        SparkSession spark = SparkSession
                .builder()
                .appName("SparkSample")
                .master("local[*]")
                .getOrCreate();
        //Create Dataset
        List<Tuple2<String,Double>> inputList = new ArrayList<Tuple2<String,Double>>();
        inputList.add(new Tuple2<String,Double>("A",1.0));
        Dataset<Row> df = spark.createDataset(inputList, Encoders.tuple(Encoders.STRING(), Encoders.DOUBLE())).toDF();
        df.show(false);
        //Java 8 style of creating Array. You can create by using for loop as well
        int[] array = IntStream.range(0, 5).toArray();
        //With Dummy Column
        Dataset<Row> df1 = df.withColumn("dummy", explode(lit(array)));
        df1.show(false);
        //Drop Dummy Column
        Dataset<Row> df2 = df1.drop(col("dummy"));
        df2.show(false);
    }
}

Below are the output of this program.

+---+---+
|_1 |_2 |
+---+---+
|A  |1.0|
+---+---+

+---+---+-----+
|_1 |_2 |dummy|
+---+---+-----+
|A  |1.0|0    |
|A  |1.0|1    |
|A  |1.0|2    |
|A  |1.0|3    |
|A  |1.0|4    |
+---+---+-----+

+---+---+
|_1 |_2 |
+---+---+
|A  |1.0|
|A  |1.0|
|A  |1.0|
|A  |1.0|
|A  |1.0|
+---+---+
user_45018040
Related