get datatype of column using pyspark

user_4082822 0
Viewed 148459

We are reading data from MongoDB Collection. Collection column has two different values (e.g.: (bson.Int64,int) (int,float) ).

I am trying to get a datatype using pyspark.

My problem is some columns have different datatype.

Assume quantity and weight are the columns 

quantity           weight
---------          --------
12300              656
123566000000       789.6767
1238               56.22
345                23
345566677777789    21

Actually we didn't defined data type for any column of mongo collection.

When I query to the count from pyspark dataframe

dataframe.count()

I got exception like this

"Cannot cast STRING into a DoubleType (value: BsonString{value='200.0'})"
pyspark apache-spark-sql apache-spark
8 Answers

For anyone else who came here looking for an answer to the exact question in the post title (i.e. the data type of a single column, not multiple columns), I have been unable to find a simple way to do so.

Luckily it's trivial to get the type using dtypes:

def get_dtype(df,colname):
    return [dtype for name, dtype in df.dtypes if name == colname][0]

get_dtype(my_df,'column_name')

(note that this will only return the first column's type if there are multiple columns with the same name)

user_16612120 
import pandas as pd
pd.set_option('max_colwidth', -1) # to prevent truncating of columns in jupyter

def count_column_types(spark_df):
    """Count number of columns per type"""
    return pd.DataFrame(spark_df.dtypes).groupby(1, as_index=False)[0].agg({'count':'count', 'names': lambda x: " | ".join(set(x))}).rename(columns={1:"type"})

Example output in jupyter notebook for a spark dataframe with 4 columns:

count_column_types(my_spark_df)

enter image description here

user_12404870

df.dtypes to get a list of (colname, dtype) pairs, ex.

[('age', 'int'), ('name', 'string')]

df.schema to get a schema as StructType of StructField, ex.

StructType(List(StructField(age,IntegerType,true),StructField(name,StringType,true)))

df.printSchema() to get a tree view of the schema, ex.

root
 |-- age: integer (nullable = true)
 |-- name: string (nullable = true)
user_31636180 
data = [('A+','good','Robert',550,3000),
  ('A+','good','Robert',450,4000),
  ('A+','bad','James',300,4000),
  ('A','bad','Mike',100,4000),
  ('B-','not bad','Jenney',250,-1)
]

columns = ["A","B","C","D","E"]
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Temp-Example').getOrCreate()
df = spark.createDataFrame(data=data, schema = columns)
df.printSchema()


# root
#  |-- A: string (nullable = true)
#  |-- B: string (nullable = true)
#  |-- C: string (nullable = true)
#  |-- D: long (nullable = true)
#  |-- E: long (nullable = true)

you can get datatype by simple code

# get datatype 
from collections import defaultdict
import pandas as pd

data_types = defaultdict(list)
for entry in df.schema.fields:
    data_types[str(entry.dataType)].append(entry.name)

pd.DataFrame(list((i,len(data_types[i])) for i in data_types) , columns = ["datatype","Nums"])

#   datatype    Nums
# 0 StringType()    3
# 1 LongType()  2
user_132097510
Related