A have a strange problem converting date read from json file in utc format. I'm Croatia (timezone Europe/Zagreb). When I open JSON file with UTC datetime values (example), depending where I open it, I get different results for same code.
Example:
df.select("start").
withColumn("start_zagreb",from_utc_timestamp(to_timestamp(col("start")),"Europe/Zagreb")).
withColumn("start_frankfurt",from_utc_timestamp(to_timestamp(col("start")),"Europe/Frankfurt")).
withColumn("start_paris",from_utc_timestamp(to_timestamp(col("start")),"Europe/Paris")).
withColumn("start_vienna",from_utc_timestamp(to_timestamp(col("start")),"Europe/Vienna")
When I run it in jupyter notebook on AWS in Europe (eu-central-1, this is in Frankfurt), I get this (all correct but Frankfurt):
+------------------------+-------------------+-------------------+-------------------+-------------------+
|start |start_zagreb |start_frankfurt |start_paris |start_vienna |
+------------------------+-------------------+-------------------+-------------------+-------------------+
|2020-08-02T01:45:00.000Z|2020-08-02 03:45:00|2020-08-02 01:45:00|2020-08-02 03:45:00|2020-08-02 03:45:00|
So all times are correct but the one for Frankfurt.
If I run same code on my laptop (timezone is Europe/Zagreb) on local spark, a get these results (all wrong but Frankfurt):
+------------------------+-------------------+-------------------+-------------------+-------------------+
|start |start_zagreb |start_frankfurt |start_paris |start_vienna |
+------------------------+-------------------+-------------------+-------------------+-------------------+
|2020-08-02T01:55:00.000Z|2020-08-02 05:55:00|2020-08-02 03:55:00|2020-08-02 05:55:00|2020-08-02 05:55:00|
I tried with from_utc_timestamp(to_timestamp(col("start")) and with from_utc_timestamp(col("start")) and that makes no difference.
Does anybody knows what's this about? It seems that if I try to convert UTC to local time in same zone I'm in, I get wrong results...
Spark version is 2.4.6.