PySpark: cast nullType field as string under struct type column

Question

I have a dataframe which has the following schema. The translation_version field under translations --> languages (no, pt,...) column is in null. I want to cast all translation_version as string. I have 17 languages under translations

root
|-- translations: struct (nullable = true)
|    |-- no: struct (nullable = true)
|    |    |-- Description: string (nullable = true)
|    |    |-- class: string (nullable = true)
|    |    |-- description: string (nullable = true)
|    |    |-- translation_version: null (nullable = true) // Want to cast as string
|    |-- pt: struct (nullable = true)
|    |    |-- Description: string (nullable = true)
|    |    |-- class: string (nullable = true)
|    |    |-- description: string (nullable = true)
|    |    |-- translation_version: null (nullable = true)
|    |-- fr: struct (nullable = true)
|    |    |-- Description: string (nullable = true)
|    |    |-- class: string (nullable = true)
|    |    |-- description: string (nullable = true)
|    |    |-- translation_version: null (nullable = true)

I tried df = df.na.fill('null') but didn't change anything. Also tried to cast with the following code

df = df.withColumn("translations", F.col("translations").cast("struct>"))

but this returned the following error

pyspark.sql.utils.ParseException: u"
mismatched input '<' expecting ':'(line 1, pos 13)

== SQL ==
struct>
-------------^^^
"

Any idea how to cast all translation_version as string for each language?

user_2593817 · Answer

this should do the trick

from pyspark.sql.functions import col, struct
from pyspark.sql.types import StructType, StructField, StringType

schema_ = StructType([StructField("Description",StringType(),True),
                      StructField("class",StringType(),True),
                      StructField("description",StringType(),True),
                      StructField("translation_version",StringType(),True)
                     ]
                    )

df_1 = (
    df
    .select("translations.*")
    .withColumn("translations", struct(
        col("fr").cast(schema).alias("fr"),
        col("pt").cast(schema).alias("pt"),
        col("no").cast(schema).alias("no")
               )
               )
    .drop("fr", "pt", "no")
)

PySpark: cast nullType field as string under struct type column

1 Answers