pyspark round function throws "Invalid argument, not a string or column" error

user_17864956 0
Viewed 89

I have an ELT process that after the dataset has been created, the code below is executed to count the rows to determine the number of partitions.

provider_out = get_provider(spark)
numofpartitions = round(provider_out.count()/10000000)

This numofpartitions variable is used later to partition the data equally when writing to the destition as shown below.

provider_out.repartition(numofpartitions).write.mode("overwrite").parquet(dest_path)

I'm running into a problem when the numofpartions variable gets calculated and throws the "Invalid argument, not a string or column: 45.0838586 of type <class 'float'>. For column literals, use 'lit', 'array', 'struct' or 'create_map' function." error. I know this can occur when the result gets passed as an object, but it's a single float value that is being rounded.

Any idea why this may be happening?

pyspark
1 Answers

It seems you've imported pyspark sql functions without an alias. Are they being imported as from pyspark.sql.functions import *? if yes, the round() from pyspark sql functions is being called instead of python's native round(). TIP - It's a good practice to import pyspark sql functions with an alias.

pyspark's round() requires a column name (col() or lit()) to process and, thus, throws an error when given an integer.

See below test.

import pyspark.sql.functions as func

round(data1_rdd.count()/10)
# 0

func.round(data1_rdd.count()/10)  # you inadvertently called this
# TypeError: Invalid argument, not a string or column: 0.2 of type <class 'float'>. For column literals, use 'lit', 'array', 'struct' or 'create_map' function.
user_82795850
Related