Spark (with python) problems with withColumn

user_12698527 0
Viewed 49

I'm learning to use python and in my project, I specifically need to use spark. (I'm confident with other programming languages [C++, Matlab, R] )

I want to read a csv file and as a first step, I wanted to convert the content of the columns (strings) in the proper types (date, int and double respectively).

Below is what I wrote

from pyspark.sql import SparkSession
from pyspark.sql.functions import udf
from pyspark.sql.types import DoubleType
from pyspark.sql.types import IntegerType
from pyspark.sql.types import DataType

spark = SparkSession.builder.appName("my_spark").getOrCreate()

# DATA PREPROCESSING


A2A_av1=spark.read.format('csv').options(header='true').load('../../Data/A2A_avail.csv')
A2A_av1.printSchema()


A2A_av1.withColumn("SUM_AVAILABLE_", A2A_av1["SUM_AVAILABLE_"].cast(DoubleType()))

A2A_av1.printSchema()

Here the output, nothing changes. The column SUM_AVAILBELS_ is still a string.

Could you tell me what I'm doing wrong? Thanks in advance

pyspark
1 Answers

Pyspark does not offer inplace editing like pandas. so you have to assign the result to make the changes effective.Spark dataframes are immutable. change your code to

A2A_av1=A2A_av1.withColumn("SUM_AVAILABLE_", A2A_av1["SUM_AVAILABLE_"].cast(DoubleType()))

This should work :-)

user_112378340
Related