Convert String to Date in PySpark (to_date)

Viewed 28

I have an AWS Glue PySpark script loading a CSV file stored in S3 to our data warehouse (MySQL). Except for one field, all other fields properly load to the warehouse. The field that is failing is a perceived date field in the CSV. Coming into the PySpark environment the root is a string. Shown here:

enter image description here

Code block using 'to_date' to convert string to date. Note in the CSV the date period is '6/1/2022'

#formatting report_period date
dataframe2.select(dataframe2.report_period,to_date(dataframe2.report_period,'MM/d/yyyy'))

Output schema still displaying report_period as a string.

enter image description here

The desired outcome is that the report_period field be converted from a string to a date.

0 Answers
Related