Converting a column to date format (DDMMMyyyy) in pyspark.I am getting whole column(date) as null

user_15233279 0
Viewed 391

While changing the format of column week_end_date from string to date, I am getting whole column as null.

from pyspark.sql.functions import unix_timestamp, from_unixtime
df = spark.read.csv('dbfs:/location/abc.txt', 
header=True)

df2 = df.select(
'week_end_date', 
from_unixtime(unix_timestamp('week_end_date', 'MM-dd-yyyy')).alias('date')
).show()

print(df2)

enter image description here

pyspark apache-spark-sql apache-spark
1 Answers

Your date format is incorrect. It should be ddMMMyy. You can also directly use to_date instead of unix timestamp functions.

import pyspark.sql.functions as F

df = spark.read.csv('dbfs:/location/abc.txt', header=True)

df2 = df.select(
    'week_end_date', 
    F.to_date('week_end_date', 'ddMMMyy').alias('date')
)

If you want the format to be transformed to MM-dd-yyyy, you can use date_format:

df2 = df.select(
    'week_end_date', 
    F.date_format(F.to_date('week_end_date', 'ddMMMyy'), 'MM-dd-yyyy').alias('date')
)
user_141657300
Related