I have a column called "StartTime" which looks like this:
and I am using the following command to only get the date and change it to int:
df = df.withColumn("detection_date_id",date_format(to_date(col("StartTime"),"M/dd/yyyy h:mm:ss a"),"yyyyMMdd").cast("int"))
Expected result would be: 20102202, however it returns only nulls. Weirdly enough this piece of code works as expected in a different ETL. Anyone know what could be the issue?