PickleException: expected zero arguments for construction of ClassDict (for numpy.dtype)

user_5199660 0
Viewed 711

I don't understand how this could be fixed, I went through on some of the questions here already, but didn't find a perfectly fitting answer.

I have a dataframe which has the following important columns: building_id, area, height.

The UDF I tried to write calculates a difference between the square root of the area and the height. It returns a value, which should be added to the dataframe.

def calculate_difference(area, height):
  # calculate the square root of the area
  import numpy as np
  nr = np.sqrt(area)
  
  # calculate the difference between the square root of the area and the height
  dif = nr - height
  
  return dif

And then I register this UDF:

calculate_differenceUDF = udf(calculate_difference)

The function works when I pass two numbers, it returns the value I expect. I want to add a new column to my dataframe, where we have a calculated value, based on the function.

display(df.withColumn("diff", calculate_differenceUDF(col("area"), col("height"))))

Then I receive this error:

PickleException: expected zero arguments for construction of ClassDict (for numpy.dtype)

I understand that I don't return maybe a correct type, but I don't see how to fix it! :)

pyspark databricks user-defined-functions
2 Answers

I think you should first convert the returned value of numpy.sqrt() to python's float type.

def calculate_difference(area, height):
  
  nr = float(np.sqrt(area))
  dif = nr - height
  return dif

then register the UDF

calculate_differenceUDF = udf(calculate_difference, FloatType())
user_94778430

Other answer correct on ensuring the appropriate datatype is returned (in this case, float). If others are still facing the same error, I also had to ensure my inputs were of the appropriate type. For example:

def calculate_difference(area, height):
  # specify input datatype
  area = float(area)
  height = float(height)
  # calculate the square root of the area
  import numpy as np
  nr = np.sqrt(area)
  
  # calculate the difference between the square root of the area and the height
  dif = nr - height
  
  return dif
user_199974730
Related