How do I get column and nested field names in pyspark?

user_5113407 0
Viewed 92

I have a pyspark df.

df.printSchema()

root
 |-- bio: string (nullable = true)
 |-- city: string (nullable = true)
 |-- company: string (nullable = true)
 |-- custom_fields: struct (nullable = true)
 |    |-- nested_field1: string (nullable = true)
 |-- email: string (nullable = true)
 |-- first_conversion: struct (nullable = true)
 |    |-- nested_field2: struct (nullable = true)
 |    |    |-- number: string (nullable = true)
 |    |    |-- state: string (nullable = true)

I would like to iterate over column and nested fields in order to get their names (just their names). I should be able to print them and get the following result:

bio
city
company
custom_fields
nested_field1
email
first_conversion
nested_field2
number
state

I can easily print the first level with:

for st in df.schema:
  print(st.name)

But how do I check for deeper levels in runtime and recursively?

pyspark python
1 Answers

dtypes will give you more details of the schema, you will have to parse it though

df.printSchema()
root
 |-- id: integer (nullable = true)
 |-- rec: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- a: integer (nullable = true)
 |    |    |-- b: float (nullable = true)

df.dtypes
# [('id', 'int'), ('rec', 'array<struct<a:int,b:float>>')]
user_34415100
Related