Get names of all columns with StringType

user_10072357 0
Viewed 640

How can I get all column names where datatype is not StringType? I have:

dt = [(10,20,50, 'Big'), 
        (4,5,20,'Small'), 
        (3,1,30,'Middle'), 
        (34,100,40,'Any') 
      ]
dtColumns = ["valA","valB", "valC","text"]
df = spark.createDataFrame(data=dt, schema = dtColumns)
    
for f in df.schema.fields:
       if f !='StringType':
          print(df.columns)

But this doesn't work. I am still receiving also StringTypes and also duplicates.

Result:
    ['valA', 'valB', 'valC', 'text']
    ['valA', 'valB', 'valC', 'text']
    ['valA', 'valB', 'valC', 'text']
    ['valA', 'valB', 'valC', 'text']
pyspark apache-spark dataframe
2 Answers

Use df.dtypes if you want string comparaison:

non_string_cols = [c for c, t in df.dtypes if t !='string']

print(non_string_cols)
user_13865510

You can compare the dataType of the field to the Spark datatype StringType, not comparing to a string.

from pyspark.sql.types import StringType

not_string = [c.name for c in df.schema.fields if not isinstance(c.dataType, StringType)]

print(not_string)
# ['valA', 'valB', 'valC']
user_141657300
Related