pyspark - converting epoc time to date

user_2504762 0
Viewed 33

I have a column with epoc time which I want to convert to date. However while trying it gives me wrong date.

>>> parsed_df.withColumn("event_date", from_unixtime(col('event_time').cast('long')).alias("event_date")).select("event_time", "event_date").limit(10).show()
+-------------+--------------------+
|   event_time|          event_date|
+-------------+--------------------+
|1614870695528|53143-03-20 21:12:08|
|1611501104538|53036-06-08 23:42:18|
|1615043979790|53148-09-15 11:43:10|
|1614654606398|53136-05-14 20:26:38|
|1612020007207|53052-11-17 18:20:07|
|1611630590580|53040-07-16 16:03:00|
|1623251369665|53408-10-15 10:14:25|
|1611026130542|53021-05-21 14:29:02|
|1611544602034|53037-10-25 10:20:34|
|1612150206612|53057-01-02 16:50:12|
+-------------+--------------------+
pyspark apache-spark-sql apache-spark
1 Answers

Dividing event_time column by 1000 should work for you. Epoch should be in seconds for from_unixtime method but in your case it's in millisecond.

parsed_df.withColumn("event_date", from_unixtime(col('event_time').cast('long')/1000).alias("event_date"))
user_87733090
Related