How do i handle Nulls in pattern matching in spark scala

user_3240790 0
Viewed 102

Here is spark shell code

 scala> val colName = "time_period_id"


 scala> val df = spark.sql("""select time_period_id from prod.demo where time_period_id = 
 202101102 """)
 df: org.apache.spark.sql.DataFrame = [time_period_id: int]

 scala>  val result = df.agg(max(colName)).head(1)
 result: Array[org.apache.spark.sql.Row] = Array([null])


 scala>     result(0).getInt(0) match {
 |       case null => 0
 |       case _ => result(0).getInt(0)
 |     }

If the result is Array([null]) then I want 0 to be returned, If the result is Array([20210110]) then i want 20210110 to be returned

But i am getting this error

 <console>:33: error: type mismatch;
 found   : Null(null)
 required: Int
         case null => 0
apache-spark scala
2 Answers

You can get your array item with lift in order to deal with it as an Option. It will return Some(null) if the element is null inside, or None if there is no element at all

result.lift(0) match {
 case Some(element) if element != null => element.getInt(0)
 case _ => 0
}

Or if you don't mind to have more rows in your pattern matching:

  result.lift(0) match {
     case Some(null) => 0
     case Some(element) => element.getInt(0)
     case _ => 0
    }

Another option is to use Try

import scala.util.Try
import scala.util.Success

Try(result(0).getInt(0)) match {
  case Success(date) => date
  case _ => 0
}
user_63783110

You could also use the isNullAt() method on result(0):

scala> val result = Array(org.apache.spark.sql.Row(null))
result: Array[org.apache.spark.sql.Row] = Array([null])

scala> result(0) match {
     | case x if x.isNullAt(0) => 0
     | case x => x.getInt(0)
     | }
res0: Int = 0
user_138418360
Related