Is it possible to override just one column type when using PySpark to read in a CSV?

user_1207596 0
Viewed 95

I'm trying to use PySpark to read in a CSV file with many columns. The inferschema option is great at inferring majority of the columns' data types. If I want to override just one of the columns types that were inferred incorrectly, what is the best way to do this?

I have this code working, but it makes PySpark import only the one column that is specified in the schema, which is not want I want.

schema = StructType() \ 
    .add("column_one_of_many", StringType(), True) 

spark.read.format('com.databricks.spark.csv') \ 
  .option('delimited',',') \ 
  .option('header','true') \ 
  .option('inferschema', 'true') \ 
  .schema(self.schema) \ 
  .load('dbfs:/FileStore/some.csv')

Is what I'm asking for even possible?

Thank you for your time and guidance :)

pyspark apache-spark python
1 Answers

Easier way would be using .withColumn and casting column_one_of_many as string.

Example

from pyspark.sql.types import *

spark.read.format('com.databricks.spark.csv') \ 
  .option('delimited',',') \ 
  .option('header','true') \ 
  .option('inferschema', 'true') \ 
  .load('dbfs:/FileStore/some.csv')\
  .withColumn("column_one_of_many",col("column_one_of_many").cast("string"))

Other way would be defining all the columns in schema then exclude the inferschema just use .schema option to read the csv file.

user_76326950
Related