How to optimize the PySpark toPandas() with type hints

user_2573069 0
Viewed 2586

I have not seen this warning in PySpark before:

The conversion of DecimalType columns is inefficient and may take a long time. Column names: [PVPERUSER] If those columns are not necessary, you may consider dropping them or converting to primitive types before the conversion.

What is the best way to handle it? Is this a parameter passed into toPandas() or do I need to type the dataframe in a particular way?

My code is a simple pyspark conversation to pandas:

df = data.toPandas()
pyspark
2 Answers

Try this:

df = data.select(data.PVPERUSER.cast('float'), data.another_column).toPandas()
user_52720150

Alternative approach when converting Spark DF to pandas DF when you want to convert many columns:

from pyspark.sql.types import FloatType
from pyspark.sql.functions import col

#find all decimal columns in your SparkDF
decimals_cols = [c for c in df.columns if 'Decimal' in str(df.schema[c].dataType)]

#convert all decimals columns to floats
for col in decimals_cols:
    df = df.withColumn(col, df[col].cast(FloatType()))

#Now you can easily convert Spark DF to Pandas DF without decimal errors
pandas_df = df.toPandas() 

#sometimes if you have spaces in the column names it will convert to lowercase 
pandas_df.columns = pandas_df.columns.str.upper()
user_110589300
Related