Convert Integer Column to Date in PySpark

user_13189547 0
Viewed 8836

I have an Integer column called birth_date in this format: 20141130

I want to convert that to 2014-11-30 in PySpark.

This converts the date incorrectly:

.withColumn("birth_date", F.to_date(F.from_unixtime(F.col("birth_date"))))

This gives an error: argument 1 requires (string or date or timestamp) type, however, 'birth_date' is of int type

.withColumn('birth_date', F.to_date(F.unix_timestamp(F.col('birth_date'), 'yyyyMMdd').cast('timestamp')))

What is the best way to convert it to the date I want?

pyspark-dataframes pyspark apache-spark-sql
2 Answers

Convert the birth_date column from Integer to String before you pass it to the to_date function:

from pyspark.sql import functions as F

df.withColumn("birth_date", F.to_date(F.col("birth_date").cast("string"), \
    'yyyyMMdd')).show()

+----------+
|birth_date|
+----------+
|2014-11-30|
+----------+
user_92389280

I hope the following code should work fine.

import pyspark.sql.functions as psf
raw_df.select('timestamp')\
.withColumn("as_date", psf.to_utc_timestamp(psf.from_unixtime(psf.col("timestamp"),'yyyy-MM-dd HH:mm:ss'),'IST')).show(20)
user_113855760
Related