Iterate the row in dataframe based on the column values in spark scala

user_18088261 0
Viewed 53

For 'N' farmers on the invoice each owner will be split 'N' times. I want the row to split as per expected output in spark scala.

source Data:

INVNUM  PRODNUM CUSTNAME    LANDOWNERS
1       1       Sonu        farmer
1       1       Ramu        farmer
1       1       Alec        owner
1       1       Prashanth   owner

Expected output:

INVNUM  PRODNUM CUSTNAME    Shareownername
1       1       Sonu        Sonu --> same name here as Sonu is Farmer
1       1       Ramu        Ramu --> same name here as Ramu is Farmer
1       1       Alec        Sonu --> split two times for Sonu and Ramu as sharewoner for Alec
1       1       Alec        Ramu
1       1       Prashanth   Sonu --> split twice Sonu and Ramu as sharewoner for Prashanth
1       1       Prashanth   Ramu
apache-spark scala
1 Answers

Owners can be joined with farmers for get split, and united by "union" with farmers:

val df = Seq(
  (1, 1, "Sonu", "farmer"),
  (1, 1, "Ramu", "farmer"),
  (1, 1, "Alec", "owner"),
  (1, 1, "Prashanth", "owner")
).toDF("INVNUM", "PRODNUM", "CUSTNAME", "LANDOWNERS")

val farmers = df.where($"LANDOWNERS" === "farmer")
val owners = df.where($"LANDOWNERS" === "owner")

val split = owners.alias("o")
  .join(farmers.alias("f"), Seq("INVNUM", "PRODNUM"))
  .select($"o.INVNUM", $"o.PRODNUM", $"o.CUSTNAME", $"f.CUSTNAME".alias("Shareownername"))


val result = farmers
  .select($"INVNUM", $"PRODNUM", $"CUSTNAME", $"CUSTNAME".alias("Shareownername"))
  .unionAll(split)

Result:

+------+-------+---------+--------------+
|INVNUM|PRODNUM|CUSTNAME |Shareownername|
+------+-------+---------+--------------+
|1     |1      |Sonu     |Sonu          |
|1     |1      |Ramu     |Ramu          |
|1     |1      |Alec     |Ramu          |
|1     |1      |Alec     |Sonu          |
|1     |1      |Prashanth|Ramu          |
|1     |1      |Prashanth|Sonu          |
+------+-------+---------+--------------+
user_26673430
Related