In PySpark how to round a timestamp value to the nearest minute?

user_2363834 0
Viewed 237

I'm trying to round a timestamp column in PySpark, I can't use the date_trunc function because it only round down the value..

For example:

Real Value              Expected Round Up/Down
2020-11-03 18:25:04 ->  2020-11-03 18:25:00
2020-11-03 18:21:44 ->  2020-11-03 18:22:00

I would like not to use pandas to do the solution..

apache-zeppelin aws-glue pyspark timestamp python
1 Answers

You can add 30 seconds to the timestamp and then truncate to the minute. This assumes your column is named 'time'

import pyspark.sql.functions as f

df = df.withColumn('time_minute', f.date_trunc('minute', f.col('time') + f.expr('INTERVAL 30 SECONDS')))

+-------------------+-------------------+                                       
|               time|        time_minute|
+-------------------+-------------------+
|2020-11-03 18:25:04|2020-11-03 18:25:00|
|2020-11-03 18:21:44|2020-11-03 18:22:00|
+-------------------+-------------------+
user_47003270
Related