how to change month name to a different language in pyspark - dataframe

user_15749660 0
Viewed 355

I am trying to create a table for "Date" on Databricks using the below configurations:

# Get date range
dateFrom = dbutils.widgets.get("date_from")
dateTo = dbutils.widgets.get("date_to")

dateDF_TESTE = spark.sql("SELECT sequence(to_date('{0}'), to_date('{1}'), interval 1 day) AS date".format(dateFrom, dateTo))\
               .select(F.explode("date").alias('DSC_DATE'))'''

But when I add columns with those data, I am only getting the information, for example month name or days of the week, in english.

I intend to change this information to another language (portuguese), but without any success. I´ve tried to use locale but it is not working.

import locale
# use user's default settings
locale.setlocale(locale.LC_ALL, 'pt_PT.utf8')
pyspark databricks dataframe settings
1 Answers

Since Spark 3.0 it is possible to use to_csv() on a single column. to_csv accepts the same parameters like the standard csv writer, so it is possible to set the locale here:

from pyspark.sql import functions as F

dateDF_TESTE.withColumn("formatted_date",
    F.to_csv(F.struct(F.col("DSC_DATE")),
        {"dateFormat": "EEEE, d 'de' MMMM 'de' yyyy", "locale": "pt", "quote":""}))\
    .show(truncate=False, n=5)

prints

+----------+------------------------------------+
|DSC_DATE  |formatted_date                      |
+----------+------------------------------------+
|2020-01-01|Quarta-feira, 1 de Janeiro de 2020|
|2020-01-02|Quinta-feira, 2 de Janeiro de 2020|
|2020-01-03|Sexta-feira, 3 de Janeiro de 2020 |
|2020-01-04|Sábado, 4 de Janeiro de 2020      |
|2020-01-05|Domingo, 5 de Janeiro de 2020     |
+----------+------------------------------------+
only showing top 5 rows
user_21298010
Related