How can I apply a function to modify column values? (Pyspark 2.4.5 - Databricks)

user_12738886 0
Viewed 110

enter image description here

My goal is map each range and map it to its mid section (15000-25000 -> 20000).

I cleaned up the data and boiled it down to this column. How can I apply the wanted transformation on the column itself\map the result to another column? I cant find any understandable data anywhere on the internet regarding pyspark..

pyspark apache-spark
2 Answers

Use Spark higher order functions aggregate from spark-2.4

Example:

df=spark.createDataFrame([('15000-25000',)],['jobsalary'])

from pyspark.sql.functions import *

df.withColumn("mid",expr('cast(aggregate(cast(split(jobsalary,"-") as array<int>),0,(acc,x) -> acc+x)/size(cast(split(jobsalary,"-") as array<int>)) as int)')).show()
#+-----------+-------+
#|  jobsalary|  mid  |
#+-----------+-------+
#|15000-25000|  20000|
#+-----------+-------+
user_76326950

For spark<2.4

df.withColumn('id',monotonically_increasing_id()).\
   withColumn('val',explode('jobsalary')).\
   groupBy('id').agg(avg('val').alias('mid')).\
   drop('id').show()

#+-----------+-------+
#|  jobsalary|  mid  |
#+-----------+-------+
#|15000-25000|  20000|
#+-----------+-------+
user_53527480
Related