How to change date format in pyspark?

datadoubts 0
Viewed 31

I have a date field in a csv. Dates can be of different formats in the same field on the same file. Like one record may have '14-Dec-2022', next record may have '21-04-2022'. How to change the date in first record(14-Dec-2022) to 14-12-2022 in Pyspark.

date-formatting pyspark apache-spark-sql
1 Answers

based on your comment, it'll be easy if it is known that there are only 2 formats present in the file when it is read in pyspark.

Here's an example based on that.

Option 1 - If you want to format in a different format

spark.sparkContext.parallelize([('14-Dec-2022',), ('14-12-2022',)]).toDF(['date_field_str']). \
    withColumn('date_field', 
               func.coalesce(func.to_date('date_field_str', 'dd-MMM-yyyy'), 
                             func.to_date('date_field_str', 'dd-MM-yyyy')
                             )
               ). \
    withColumn('date_field_formatted', func.date_format('date_field', 'dd-MM-yyyy')). \
    show()
# you can change the output format used within `date_field_formatted`'s `withColumn`.

# +--------------+----------+--------------------+
# |date_field_str|date_field|date_field_formatted|
# +--------------+----------+--------------------+
# |   14-Dec-2022|2022-12-14|          14-12-2022|
# |    14-12-2022|2022-12-14|          14-12-2022|
# +--------------+----------+--------------------+

Option 2

spark.sparkContext.parallelize([('14-Dec-2022',), ('14-12-2022',)]).toDF(['date_field_str']). \
    withColumn('date_field_formatted', 
               func.coalesce(func.date_format(func.to_date('date_field_str', 'dd-MMM-yyyy'), 'dd-MM-yyyy'), 
                             func.col('date_field_str')
                             )
               ). \
    show()

# +--------------+--------------------+
# |date_field_str|date_field_formatted|
# +--------------+--------------------+
# |   14-Dec-2022|          14-12-2022|
# |    14-12-2022|          14-12-2022|
# +--------------+--------------------+
samkart0
Related