Union operation on similar datasets

user_15622984 0
Viewed 51

I want to perform the union operation on small sized datasets which they have the same structure (same name & type of columns) in order to get one big dataset. How to do this in JAVA / SPARK ?

PS : i tried with union() but this method accepts only one dataset and not a list of datasets to union which what i want. THanks

apache-spark
1 Answers

You can union the datasets in a loop one by one:

Dataset<Long> ds1 = spark.createDataset(Arrays.asList(1L,2L,3L), Encoders.LONG());
Dataset<Long> ds2 = spark.createDataset(Arrays.asList(10L,20L,30L), Encoders.LONG());
Dataset<Long> ds3 = spark.createDataset(Arrays.asList(100L,200L,300L), Encoders.LONG());
List<Dataset<Long>> dss = new ArrayList<>(Arrays.asList(ds1, ds2, ds3));

Dataset<Long> result = dss.remove(0);
for( Dataset<Long> ds: dss) {
     result = result.union(ds);
}
user_21298010
Related