Getting SparkUpgrade exception while trying to convert string to unix_timestamp datatype in pyspark dataframe

user_15735333 0
Viewed 96

I am using spark3.1.1 and trying to convert string-datatype to unix_timestamp datatype using the below code.

pattern = 'yyyy/MM/dd hh:mm:ss aa'
file_new = file1.withColumn('Incident_DateTime', unix_timestamp(file1['Incident_DateTime'], pattern).cast('timestamp'))  
file_new.select('Incident_DateTime').show(5, False)

Getting error on all actions -> select/display/show. PFB snapshot and help

org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: Fail to recognize 'yyyy/MM/dd hh:mm:ss aa' pattern in the DateTimeFormatter. 1) You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0. 2) You can form a valid datetime pattern with the guide from https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html

pyspark apache-spark casting
1 Answers

From the link in your question:

"am-pm: This outputs the am-pm-of-day. Pattern letter count must be 1.",

so the pattern should be with a single 'a':

'yyyy/MM/dd hh:mm:ss a'
user_47003270
Related