Convert none type to datetime using parser.parse

user_16459035 0
Viewed 320

I'm using the following function to parse string to date in PySpark

func = udf(lambda x: parser.parse(x), DateType())

My date format is:

"22-Jan-2021 00:00"

Althought this function does not work with None types, I have the following Spark data frame

date
----
"22-Jan-2021 00:00"
""
"10-Feb-2020 14:00"

When I apply my func to the date column I got an error on the second line of DF saying that it can't parse NoneType. Any tips to solve this problem using PySpark and the above func ?

An MVCE:

date = None
date_parsed = parser.parse(date)
python-datetime pyspark python
1 Answers

It seems like you could just use the to_timestamp function.

ex.

df.show()
+---+-----------------+
| id|             date|
+---+-----------------+
|  1|22-Jan-2021 00:00|
|  2|             null|
|  3|10-Feb-2020 14:00|
+---+-----------------+

You can simply use the following code to convert the string in the date column to a timestamp type.

from pyspark.sql import functions

df = df.withColumn("date", functions.to_timestamp("date", "dd-MMM-yyyy HH:mm"))

df.show()
+---+-------------------+
| id|               date|
+---+-------------------+
|  1|2021-01-22 00:00:00|
|  2|               null|
|  3|2020-02-10 14:00:00|
+---+-------------------+

You can also verify that the conversion is done correctly with df.schema

print(df.schema)
StructType(List(StructField(id,StringType,true),StructField(date,TimestampType,true)))
user_147222970
Related