I am trying to read the data from source and insert data into another Hive table, but all timestamp column's values have changed (-4 hour). Why it changes the timestamp?
This is how the source looks like:
ID DATE
1, 2019-05-14 00:00:00
This is how Hive table looks like after INSERT:
ID DATE
1, 2019-05-13 20:00:00
I don't have any idea why all TIMESTAMP columns have changed to -4 HOURS and hope that somebody already faced this issue?
I have tried a couple of options which are stated below.
- Using
conf.set("spark.sql.session.timeZone", "EDT")in code - Cast the datatype to StringType and then load it to Timestamp column
- Used
option("timestampFormat", "yyyy-MM-dd hh:mm:ss")before loading data - used
TimeZone.setDefault(TimeZone.getTimeZone("EDT"))before loading
I know that it is happening because of the timezone difference. But how to resolve this?