pySpark: How can I get all element names in structType in arrayType column in a dataframe?

user_15278499 0
Viewed 1701

I have a dataframe that looks something like this:

 |-- name: string (nullable = true)
 |-- age: string (nullable = true)
 |-- job: string (nullable = true)
 |-- hobbies: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- favorite: string (nullable = true)
 |    |    |-- non-favorite: string (nullable = true)

And I'm trying to get this information:

['favorite', 'non-favorite']

However, the only closest solution I found was using the explode function with withColumn, but it was based on the assumption that I already know the names of the elements. But What I want to do is, without knowing the element names, I want to get the element names only with the column name, in this case 'hobbies'. Is there a good way to get all the element names in any given column?

pyspark apache-spark-sql apache-spark python
2 Answers

For a given dataframe with this schema:

df.printSchema()

root
 |-- hobbies: array (nullable = false)
 |    |-- element: struct (containsNull = false)
 |    |    |-- favorite: string (nullable = false)
 |    |    |-- non-favorite: string (nullable = false)

You can select the field names of the struct as:

struct_fields = df.schema['hobbies'].dataType.elementType.fieldNames()

# output: ['favorite', 'non-favorite']
user_141657300

pyspark.sql.types.StructType.fieldnames should get you what you want.

fieldNames()

Returns all field names in a list.

>>> struct = StructType([StructField("f1", StringType(), True)])
>>> struct.fieldNames()
['f1']

So in your case something like

dataframe.hobbies.getItem(0).fieldnames()
user_94371580
Related