How do you fix type mismatch in try-catch blocks?

Question

I've written a try-catch construct which attempts to read parquet files based on an S3 path (paths) obtained from an earlier function. The try-catch construct will change the path if the try block fails and will then try again using the new path:

val rawDF: DataFrame = {
      try {
        spark.read.format("parquet").load(paths)
      } catch {
        case NonFatal(e) =>
        Thread.sleep(3600000)
        val hour = LocalDateTime.now().format(hourParser)
        val date = LocalDateTime.now().format(dateParser)
        val paths = f"s3a://twitter-kafka-app/processed-data/date=$date/hour=$hour/*"
        try {
          spark.read.format("parquet").load(paths)
        } catch {
          case NonFatal(e) => None
          print("No path found.")
          }
        }
    }
    rawDF.show()

All of these blocks work fine except the final catch block, which causes type mismatch problems because it returns Unit. I've had to specify rawDF as a DataFrame type in order for rawDF.show() to work. I've tried adding 'null' because I was under the assumption that this is how you get around this kind of problem, but it just returns NullPointerException even though I know that the first try block should successfully return the dataframe.

Is there an easy way around this that I'm missing? Thanks.

Alin Gabriel Arhip · Answer

Much of this has already been said, but using a Try is the better option to get rid of the try-catch blocks. Also it signals the operation might fail, but we're not really interesting in what it throws as long as it's non-fatal and we can recover from. Either can also be used but it's semantically different.

The only question remains is how will you treat the failure. I would return an empty DataFrame. Something like:

def tryOfRawDF(paths: String): Try[DataFrame] = Try {
  spark.read.format("parquet").load(paths)
}

val rawDF: DataFrame = tryOfRawDF(paths).getOrElse {
  Thread.sleep(3600000)
  val hour  = LocalDateTime.now().format(hourParser)
  val date  = LocalDateTime.now().format(dateParser)
  val paths = f"s3a://twitter-kafka-app/processed-data/date=$date/hour=$hour/*"
  tryOfRawDF(paths).getOrElse {
    println("No path found.")
    DataFrame.empty
  }
}
rawDF.show()

How do you fix type mismatch in try-catch blocks?

1 Answers