Pyspark: Convert column to lowercase

user_3491648 0
Viewed 100949

I want to convert the values inside a column to lowercase. Currently if I use the lower() method, it complains that column objects are not callable. Since there's a function called lower() in SQL, I assume there's a native Spark solution that doesn't involve UDFs, or writing any SQL.

pyspark
3 Answers

Import lower alongside col: 

from pyspark.sql.functions import lower, col

Combine them together using lower(col("bla")). In a complete query: 

spark.table('bla').select(lower(col('bla')).alias('bla'))

which is equivalent to the SQL query 

SELECT lower(bla) AS bla FROM bla

To keep the other columns, do

spark.table('foo').withColumn('bar', lower(col('bar')))

Needless to say, this approach is better than using a UDF because UDFs have to call out to Python (which is a slow operation, and Python itself is slow), and is more elegant than writing it in SQL.

user_34916480

You can use a combination of concat_ws and split

from pyspark.sql.functions import *

df.withColumn('arr_str', lower(concat_ws('::','arr'))).withColumn('arr', split('arr_str','::')).drop('arr_str')
user_8106870

Another approach which may be a little cleaner:

import pyspark.sql.functions as f

df.select("*", f.lower("my_col"))

this returns a data frame with all the original columns, plus lowercasing the column which needs it.

user_45135090
Related