Why doesn't from_utc_timstamp throw an error when passed a malformed timezone string in Spark?

user_12017253 0
Viewed 365

When calling the from_utc_timestamp function in spark 2.4.3 no error is thrown if I pass in a malformed timezone string. Instead, it just defaults to UTC, which is counter to my expectations, and also seems likely to cause mistakes to go unnoticed. Is this intentional, or is this a bug in Spark?

See example below:

scala> val df =  Seq(("2020-01-01 00:00:00")).toDF("date")
df: org.apache.spark.sql.DataFrame = [date: string]

scala> df.show()

// Not a real timezone obviously. Just gets treated like UTC.
scala> df.withColumn("est", from_utc_timestamp(col("date"), "not_a_real_timezone")).show()
+-------------------+-------------------+
|               date|                est|
+-------------------+-------------------+
|2020-01-01 00:00:00|2020-01-01 00:00:00|
+-------------------+-------------------+

// Typo in EST5PDT, so still not a real timezone. Also defaults to UTC, which makes it
// very easy to miss this mistake.
scala> df.withColumn("est", from_utc_timestamp(col("date"), "EST5PDT")).show()
+-------------------+-------------------+
|               date|                est|
+-------------------+-------------------+
|2020-01-01 00:00:00|2020-01-01 00:00:00|
+-------------------+-------------------+

// EST8EDT is a real timezone, so this works as expected.
scala> df.withColumn("est", from_utc_timestamp(col("date"), "EST5EDT")).show()
+-------------------+-------------------+
|               date|                est|
+-------------------+-------------------+
|2020-01-01 00:00:00|2019-12-31 19:00:00|
+-------------------+-------------------+
apache-spark scala
2 Answers

The from_utc_timestamp uses DateTimeUtils from org.apache.spark.sql.catalyst.util. To get the timezone they use getTimeZone method. This method generally does not throw an issue.

  • This might be a JVM thing, where JVM tries to avoid dependence on the system's default locale, charset and timezone
  • This might be a Spark issue to be logged in Jira

But looking at code of other people, they do have a setup of checking first with:

import java.util.TimeZone

...

if (!TimeZone.getAvailableIDs().contains(tz)) {
  throw new IllegalStateException(s"The setting '$tz' is not recognized as known time zone")
}

EDIT1: Just found out it is a "feature". It is in the migration guide for 3.0.0

In Spark version 2.4 and earlier, invalid time zone ids are silently ignored and replaced by GMT time zone, for example, in the from_utc_timestamp function. Since Spark 3.0, such time zone ids are rejected, and Spark throws java.time.DateTimeException.

https://spark.apache.org/docs/3.0.0-preview/sql-migration-guide.html

user_55941800

from_utc_timestamp calls DateTimeUtils.fromUTCTime. 

def fromUTCTime(time: SQLTimestamp, timeZone: String): SQLTimestamp = {
  convertTz(time, TimeZoneGMT, getTimeZone(timeZone))
}

To transform the timezone string into an TimeZone object the function calls getTimeZone and here the JDK's TimeZone.getTimeZone is called to convert the timezone string into an actual timezone object. The Javadoc of this method states that the method returns

the specified TimeZone, or the GMT zone if the given ID cannot be understood

In your the case of EST5PDT no time zone for this string can be found and thus the return value is GMT. As result the the SQLTimestamp is converted from GMT to GMT which means it stays unchanged.

user_21298010
Related