PySpark to_utc_timestamp return the same time

user_4692937 0
Viewed 5179

I have a table which has a datetime in string type. I want to convert it into UTC timestamp. My local time zone is CDT. I first convert datetime into timestamp. 

table = table.withColumn('datetime_dt', unix_timestamp(col('datetime'), "yyyy-MM-dd HH:mm:ss").cast("timestamp"))

Then, I try to convert this timestamp column into UTC time.

table = table.withColumn('datetime_UTC',  to_utc_timestamp(table.datetime_dt, 'CDT'))

I also try this

table = table.withColumn('datetime_UTC',  to_utc_timestamp(col('datetime_dt'), 'CDT'))

But it returns the same results. Here are some examples.

------------------------------------------------------------------
|   datetime         |     datetime_dt     |    datetime_UTC     |
------------------------------------------------------------------
|2019-01-01 00:49:00 | 2019-01-01 00:49:00 | 2019-01-01 00:49:00 |
------------------------------------------------------------------
|2019-01-01 02:06:00 | 2019-01-01 02:06:00 | 2019-01-01 02:06:00 |
------------------------------------------------------------------
|2019-01-02 05:15:00 | 2019-01-02 05:15:00 | 2019-01-02 05:15:00 |
------------------------------------------------------------------

Why does it give the same time without any conversion? I import to_utc_timestamp from pyspark.sql.functions.

pyspark apache-spark hive timezone timestamp
1 Answers

Specify timezone as CST(or) America/Chicago instead of CDT, We don't have to explicitly mention about daylight saving time.

  • Spark internally calculates and adds +5:00/+6:00 based on the date.

Example:

df.show()
#+-------------------+
#|                 dt|
#+-------------------+
#|2019-01-01 00:49:00|
#|2019-11-01 00:49:00|
#+-------------------+

df.withColumn('datetime_UTC',  to_utc_timestamp(col('dt'), 'CST')).show(10,False)

#or

df.withColumn('datetime_UTC',  to_utc_timestamp(col('dt'), 'America/Chicago')).show(10,False)

#+-------------------+---------------------+
#|dt                 |datetime_UTC         |
#+-------------------+---------------------+
#|2019-01-01 00:49:00|2019-01-01 06:49:00.0|
#|2019-11-01 00:49:00|2019-11-01 05:49:00.0|
#+-------------------+---------------------+
user_76326950
Related