I try to convert a column from string to timestamp with this code
from pyspark.sql.functions import unix_timestamp
(sc
.parallelize([Row(dt='2017-01-23T08:12:39.929+01:00')])
.toDF()
.withColumn("parsed", unix_timestamp("dt", "yyyy-MM-ddThh:mm:ss")
.cast("double")
.cast("timestamp"))
.show(1, False))
but I get null
+-----------------------------+------+
|dt |parsed|
+-----------------------------+------+
|2017-01-23T08:12:39.929+01:00|null |
+-----------------------------+------+
why ?