AttributeError: Can't get attribute '_fill_function' on <module 'pyspark.cloudpickle' from 'pyspark/cloudpickle/__init__.py'>

user_1453247 0
Viewed 936

While executing pyspark code from a script. Getting following error while df.show().

from pyspark.sql.types import StructType,StructField, StringType, IntegerType
data = [("James","","Smith","36636","M",3000),
    ("Michael","Rose","","40288","M",4000)]

schema = StructType([
    StructField("firstname",StringType(),True),
    StructField("middlename",StringType(),True),
    StructField("lastname",StringType(),True),
    StructField("id", StringType(), True),
    StructField("gender", StringType(), True),
    StructField("salary", IntegerType(), True)
  ])
 
df = spark.createDataFrame(data=data,schema=schema)
df.printSchema()
df.show(truncate=False)



AttributeError: Can't get attribute '_fill_function' on <module 'pyspark.cloudpickle' from '/Users/amijha0/Applications/apache-spark/spark-3.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/cloudpickle/__init__.py'>
pyspark apache-spark python
2 Answers

The issue is because of pyspark version. I checked the installed modules using pip freeze

$ python -m pip freeze | grep pyspark
pyspark==3.0.0

Path says it is using Spark-3.1.1 and i am using pyspark==3.0.0 which in not having "_fill_function" function on pyspark.cloudpickle module. Hence the AttributeError.

For solution, I upgraded the pyspark version

 python -m pip install --upgrade pyspark==3.1.1 --use-feature=2020-resolver
user_14532470

I had the same issue and fixed it by upgrading pyspark.

$ python -m pip install --user --upgrade pyspark

The version I had was pyspark 3.0.0, and after the upgrade command above, it was pyspark 3.2.1

Problem solved

user_164891610
Related