Output Timestamp columns value is changing during write operation in Scala Spark?

Viewed 146

I am trying to read the data from source and insert data into another Hive table, but all timestamp column's values have changed (-4 hour). Why it changes the timestamp?

This is how the source looks like:

ID        DATE
1,   2019-05-14 00:00:00

This is how Hive table looks like after INSERT:

ID        DATE
1,   2019-05-13 20:00:00

I don't have any idea why all TIMESTAMP columns have changed to -4 HOURS and hope that somebody already faced this issue?

I have tried a couple of options which are stated below.

  1. Using conf.set("spark.sql.session.timeZone", "EDT") in code
  2. Cast the datatype to StringType and then load it to Timestamp column
  3. Used option("timestampFormat", "yyyy-MM-dd hh:mm:ss") before loading data
  4. used TimeZone.setDefault(TimeZone.getTimeZone("EDT")) before loading

I know that it is happening because of the timezone difference. But how to resolve this?

0 Answers
Related