Timestamp converted to UTC while loading to parquet

Question

I am loading data to parquet through spark.

dataFrame.write.parquet(path)

My data have a timestamp column while writing to parquet it is actually converting timestamp to UTC timezone and then storing.

actual time ------- 2020-10-21 00:00:00.000
UTC time--------- 2020-10-21T05:30:00.000+05:30

I see spark conf is set to UTC timezone.spark.sql.session.timeZone Is there any way to turn off this conversion?

I want to load timestamp as is without converting it to any other timezone. how do i do that?

user_6127075 · Answer

See documentation here:

https://databricks.com/blog/2020/07/22/a-comprehensive-look-at-dates-and-timestamps-in-apache-spark-3-0.html

When writing timestamp values out to non-text data sources like Parquet, the values are just instants (like timestamp in UTC) that have no time zone information. If you write and read a timestamp value with a different session time zone, you may see different values of the hour, minute, and second fields, but they are the same concrete time instant.

Timestamp converted to UTC while loading to parquet

1 Answers