i have a dataframe with a "created_at" column that contains dates that look like this:
Mon Jan 27 17:16:46 +0000 2020
and i want to convert this column dates to:
2020-01-27
how can i do this in pyspark?
how to convert a column with dates(string) to a date only in pyspark
Viewed 47
1 Answers
Refer to Jave SimpleDate Format for more details regarding datetime characters.
df.show(truncate=False) #sample dataframe
#+------------------------------+
#|created_at |
#+------------------------------+
#|Mon Jan 27 17:16:46 +0000 2020|
#+------------------------------+
from pyspark.sql import functions as F
df.withColumn("created_at", F.to_date("created_at","EEE MMM dd HH:mm:ss +SSSS yyyy")).show()
#+----------+
#|created_at|
#+----------+
#|2020-01-27|
#+----------+
Related