AttributeError: 'NoneType' object has no attribute 'upper'

Question

I have a PySpark Dataframe (df) and was trying to add a column (capital_names) that would be an existing column (names) with names converted into capital letters. I have done it as follows:

def capital(text):
    return text.upper()

udf_capital = udf(capital,StringType())

df2 = df.withColumn("capital_names",udf_capital("names"))

df2.show()

When running the code, I get the an error when I try to show the resulting dataframe:

AttributeError: 'NoneType' object has no attribute 'upper'

However, I have checked my dataframe's schema and column "names" is StringType, how could I solve this problem? Thanks in advance!

user_14165730 · Answer

There are nulls in the column. Try to add a check in the UDF to catch that.

def capital(text):
    if text is not None:
        return text.upper()
    else:
        return None

AttributeError: 'NoneType' object has no attribute 'upper'

1 Answers