Forward fill pyspark to scala

user_12711660 0
Viewed 34

Got the following pyspark code how can I change it to adapt it to scala. Doing forwards and backwards fill on missing data

import pyspark.sql.functions as F
from pyspark.sql import Window

df = spark.createDataFrame([
    ('d1',None),
    ('d2',10),
    ('d3',None),
    ('d4',30),
    ('d5',None),
    ('d6',None),
],('day','temperature'))

w_forward = Window.partitionBy().orderBy('day').rowsBetween(Window.unboundedPreceding,Window.currentRow)
w_backward = Window.partitionBy().orderBy('day').rowsBetween(Window.currentRow,Window.unboundedFollowing)
df.withColumn('fill_forward',F.last('temperature',ignorenulls=True).over(w_forward))\
  .withColumn('fill_both',F.first('fill_forward',ignorenulls=True).over(w_backward)).show()
pyspark apache-spark scala
1 Answers

Here:

case class Day(day: String, temperature: Option[Int])

import org.apache.spark.sql.expressions.Window
import spark.implicits._
import org.apache.spark.sql.functions.{last, first}

val df = spark
  .createDataFrame[Day](
    Seq(
      Day("d1", None),
      Day("d2", Some(10)),
      Day("d3", None),
      Day("d4", Some(30)),
      Day("d5", None),
      Day("d6", None)
    )
  )

val wForward = Window
  .partitionBy()
  .orderBy($"day")
  .rowsBetween(Window.unboundedPreceding, Window.currentRow)

val wBackWard = Window
  .partitionBy()
  .orderBy($"day")
  .rowsBetween(Window.currentRow, Window.unboundedFollowing)

df.withColumn(
  "fill_forward",
  last($"temperature", ignoreNulls = true).over(wForward)
).withColumn(
  "fill_both",
  first("fill_forward", ignoreNulls = true).over(wBackWard)
).show()

Easy, isn´t it?

The main difference is that you can use a case class if you want to avoid setting explicitly the df schema using Row.

user_68021560
Related