Error when trying to specify schema for loading a CSV using pyspark

user_7398259 0
Viewed 233

I am trying to build a schema to load the contents of a file which contains the following things-

  1. movie_id, String
  2. movie_name, String
  3. plot, String
  4. genre, an array of strings

Here is a sample- enter image description here

Here is my schema definition-

customSchema = types.StructType(types.ArrayType(
                         types.StructField("movie_id", types.StringType(), True),
                         types.StructField("movie_name", types.StringType(), True),
                         types.StructField("plot", types.StringType(), True),
                         types.StructField('genre', 
                         types.ArrayType(types.StringType()), True),
))

Here is the error that I am getting

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-290-10452c69a6ff> in <module>()
        3   types.StructField("movie_name", types.StringType(), True),
        4   types.StructField("plot", types.StringType(), True),
  ----> 5   types.StructField('genre', types.ArrayType(types.StringType()), True),
        6 ))

TypeError: __init__() takes from 2 to 3 positional arguments but 5 were given
pyspark-dataframes pyspark apache-spark-sql
1 Answers

StructType() is a list of StructField()s.

from pyspark.sql.types import *  

customSchema= StructType([StructField("movie_id", IntegerType()),
                        StructField("movie_name", StringType()),
                        StructField("plot", StringType()),
                          StructField("genre", ArrayType(StringType()))])

If genre string column looks like "['Indie', 'Drama', 'Action']", try this to convert to array of strings:

from pyspark.sql import functions as F
df.withColumn("genre", F.split(F.regexp_replace("genre", "\[|]| |'", ""),",")).show(truncate=False)

#+----------------------+
#|genre                 |
#+----------------------+
#|[Indie, Drama, Action]|
#+----------------------+
user_101073890
Related